Module 4: Classification and logistic regression · Lesson 4.5 · about 25 min
Accuracy is not enough: precision, recall and F1
When one class is rare, accuracy lies. The confusion matrix counts the four kinds of right and wrong answers, and precision, recall and F1 sum them up. Move the threshold on a screening test and watch the trade-off.
Module 4 · about 25 minutes · read first: 4.1. You’ll be able to fill in a confusion matrix, compute accuracy, precision, recall, specificity and F1, and choose the right one for a job.
1. Why this matters
Most interesting classification problems are imbalanced: one class is rare. Fraud, disease, machine failure and spam are all rare. A model that always says “no” then looks highly accurate and is worthless. Every serious report of a classifier gives the confusion matrix or the scores built from it, and picking the wrong score is one of the commonest mistakes in data work.
2. The confusion matrix
Compare each prediction with the truth. There are four possible outcomes, like a smoke alarm:
| Predicted positive | Predicted negative | |
|---|---|---|
| Actually positive | True positive (TP): caught | False negative (FN): missed |
| Actually negative | False positive (FP): false alarm | True negative (TN): rightly left alone |
A true positive is the alarm going off in a real fire. A false positive is the alarm going off for burnt toast. A false negative is a real fire with a silent alarm. Statisticians also call a false positive a type I error and a false negative a type II error.
Eight emails, true labels. Emails 1, 2 and 3 are spam; 4 to 8 are normal.
The filter's guesses. It flags emails 1, 2 and 5 as spam.
Sort into the four boxes. Caught spam (TP): 1 and 2. False alarm (FP): 5. Missed spam (FN): 3. Rightly left alone (TN): 4, 6, 7, 8.
Score it. Accuracy = (2 + 4) ÷ 8 = 75 %. Precision = 2 ÷ (2 + 1) = 0.67. Recall = 2 ÷ (2 + 1) = 0.67.
What it means. When the filter says "spam", it is right two times in three. And it catches two spam emails in every three.
| Predicted: has it | Predicted: hasn't | |
|---|---|---|
| Really has it | TP | FN |
| Really hasn't | FP | TN |
3. The scores
| Score | Formula | Question it answers |
|---|---|---|
| Accuracy | (TP + TN) / all | What share of all answers are right? |
| Precision | TP / (TP + FP) | When it says “positive”, how often is it right? |
| Recall (sensitivity, true positive rate) | TP / (TP + FN) | Of the real positives, what share does it catch? |
| Specificity (true negative rate) | TN / (TN + FP) | Of the real negatives, what share does it leave alone? |
| F1 | 2 × precision × recall / (precision + recall) | One number that is high only when both precision and recall are |
Precision and recall pull against each other through the threshold. Lower the threshold and you flag more: you catch more real cases (recall up) and raise more false alarms (precision down). Raise it and the reverse happens.
- Precision
- Of everything the model flagged, the share that was really positive. High precision means few false alarms.
- Recall
- Of all the real positives, the share the model flagged. High recall means few misses.
- F1
- A blend of the two that is dragged down by whichever is worse.
- Physical meaning
- A fishing net. Recall is the share of the fish in the lake you catch; precision is the share of your catch that is fish rather than old boots and weed. A bigger net catches more fish and more boots.
Which to use?
- False alarms are costly (a spam filter hiding real mail, a fraud block on a good customer): precision.
- Misses are costly (disease screening, safety faults): recall.
- Both matter and the positive class is rare: F1, or better, look at precision and recall side by side.
- Classes balanced and both errors cost the same: accuracy is fine.
“A test that is 95 % accurate means a positive result is 95 % likely to be right.” Not when the condition is rare. Test 10,000 people for something 1 in 100 have, with a test that catches 90 % of cases and wrongly flags 5 % of healthy people: 90 true positives, but 495 false ones. Only about 15 % of positives are real. The many healthy people produce most of the alarms.
Maths and principle
Principle: each score is a ratio of confusion-matrix counts, chosen to answer one question.
F1 is the harmonic mean of precision and recall :
The harmonic mean is dragged down by the smaller value, so F1 is high only when both are high. Precision 1.0 with recall 0.02 has an ordinary average of 0.51 but an F1 of only 0.04.
Base rates matter: with a disease affecting 1 in 100 people, a test with 90 % recall and 95 % specificity flags 0.9 true cases and 4.95 false ones per 100 people tested. Its precision is only .
Try it: TP = 40, FP = 10, FN = 40, TN = 910. Find accuracy, precision, recall and F1. (Accuracy 950/1000 = 95 %; precision 40/50 = 0.8; recall 40/80 = 0.5; F1 = 2(0.8)(0.5)/1.3 ≈ 0.62.)
4. Worked example
Made-up numbers. A factory model predicts which of 2,000 parts are faulty; 50 really are.
| Threshold | TP | FP | FN | TN | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|
| 0.8 | 20 | 2 | 30 | 1,948 | 0.91 | 0.40 | 0.56 |
| 0.5 | 35 | 15 | 15 | 1,935 | 0.70 | 0.70 | 0.70 |
| 0.2 | 46 | 120 | 4 | 1,830 | 0.28 | 0.92 | 0.43 |
All three have accuracy between 94 % and 99 %, so accuracy can’t choose. A faulty part reaching a customer costs $500; checking a part by hand costs $5. Cost at 0.8: 30 × 500 + 22 × 5 = $15,110. At 0.5: 15 × 500 + 50 × 5 = $7,750. At 0.2: 4 × 500 + 166 × 5 = $2,830. The cheapest choice is the low threshold, the one with the worst F1. Let the real costs decide.
5. Practice
- A model has precision 0.9 and recall 0.3. In words, what does it do?
- Compute F1 for precision 0.6 and recall 0.6.
- Why is accuracy misleading for fraud detection?
Answers
- When it flags something it is nearly always right, but it misses most real cases. It is too cautious.
- 0.6 (when P = R, F1 equals them).
- Fraud is rare, so predicting “no fraud” for everything scores very high accuracy while catching nothing.
- The confusion matrix counts true positives, false positives, false negatives and true negatives.
- Precision = TP ÷ (TP + FP): how trustworthy a flag is. Recall = TP ÷ (TP + FN): how many real cases are caught.
- Moving the threshold trades precision against recall.
- With rare classes accuracy misleads, and even a good test can give mostly false alarms.
- When the costs of each mistake are known, choose the threshold that makes the total cost smallest.
Try it on your own data: paste your counts or your labels and scores into the confusion matrix tool to get every score at once.
New words: confusion matrix, true and false positives and negatives, precision, recall or sensitivity, specificity, F1, imbalanced classes, base rate.
Check yourself
Sources
- James, Witten, Hastie and Tibshirani (2021), An Introduction to Statistical Learning, 2nd edition, Springer, Section 4.4.2 (confusion matrix, sensitivity and specificity) https://www.statlearning.com/
- scikit-learn user guide: Metrics and scoring, classification metrics https://scikit-learn.org/stable/modules/model_evaluation.html
- Saito and Rehmsmeier (2015), The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets, PLOS ONE 10(3): e0118432 https://doi.org/10.1371/journal.pone.0118432
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.