Home / Learn / Data & AI / ML foundations / Lesson 4.6
All lessons

Module 4: Classification and logistic regression · Lesson 4.6 · about 25 min

Thresholds, ROC curves and AUC

The ROC curve shows every threshold at once: true positive rate against false positive rate. The area under it, AUC, scores how well a model ranks positives above negatives. Trace the curve and change how well the groups separate.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 4 · about 25 minutes · read first: 4.5. You’ll be able to draw an ROC curve from scores, read AUC, and pick a threshold from the curve.

The ROC curve shows every possible threshold at once, and the area under it is simply the chance that the model scores a real case above a non-case.

1. Why this matters

Precision and recall change with the threshold, so one model gives many sets of scores. To compare models fairly, before anyone has picked a threshold, we need a summary over all of them. The ROC curve and the area under it (AUC) are that summary. You’ll meet them in nearly every report on a medical test, credit score or fraud model. (ROC stands for “receiver operating characteristic”, a name left over from radar operators in the Second World War.)

2. Building the curve

For each possible threshold, work out two rates:

  • True positive rate (TPR) = recall = TP / (TP + FN): the share of real positives caught.
  • False positive rate (FPR) = FP / (FP + TN): the share of real negatives wrongly flagged. (It equals 1 − specificity.)

Plot TPR going up against FPR going across. A threshold above every score flags nothing: the point (0, 0). A threshold below every score flags everything: (1, 1). The thresholds in between trace a path from one corner to the other.

Sort by score. Six patients: three have the condition (P), three don't (N). With the threshold above everyone, nobody is flagged: point (0, 0).

Lower to 0.80. The two top scores are both P. TPR = 2/3, FPR still 0. The curve shoots straight up: that is what a good model does first.

Lower to 0.70. This one is an N: a false alarm. FPR = 1/3. The curve steps right.

Lower to 0.55. Another P: all three cases caught, one false alarm. The point (0.33, 1).

Flag everyone. The last two are N, so the curve runs across to (1, 1). The shaded area is 8/9 ≈ 0.89: the AUC.

Try it: trace the ROC curve
Predicted: has itPredicted: hasn't
Really has itTP FN
Really hasn'tFP TN
accuracy
precision
recall (sensitivity)
F1
AUC
Every threshold gives one point on the ROC curve: the share of real cases caught (true positive rate) against the share of healthy people wrongly flagged (false positive rate). Sliding the threshold walks along the curve. The area under it (AUC) sums up the model across all thresholds: 0.5 is a coin toss, 1 is perfect separation. Make the groups overlap more and the curve sinks towards the diagonal.
  • A perfect model goes straight up to (0, 1), then across: every positive scores above every negative. AUC = 1.
  • Random guessing follows the diagonal. AUC = 0.5.
  • Real models bow towards the top-left corner. The more the two groups overlap, the closer the curve hugs the diagonal.

3. What AUC means

AUC has a neat everyday meaning: it is the chance that a randomly picked positive gets a higher score than a randomly picked negative. So it measures how well the model ranks cases, whatever threshold you choose later. Rough guide: 0.5 to 0.7 weak, 0.7 to 0.8 fair, 0.8 to 0.9 good, above 0.9 excellent (and above 0.99, check for leakage, Lesson 7.4).

TPR
True positive rate: the share of real cases the model catches at a given threshold.
FPR
False positive rate: the share of non-cases it wrongly flags at that threshold.
AUC
Area under the ROC curve, from 0 to 1.
Physical meaning
A talent show judge. Pick one real talent and one dud at random: AUC is how often the judge scores the talent higher. It doesn't care where the "pass" line is drawn.

4. Choosing a threshold from the curve

Every point on the curve is a threshold you could pick. Two common ways:

  • Costs: choose the point where the total cost (missed positives × their cost + false alarms × theirs) is lowest, as in Lesson 4.5.
  • Youden’s J: with no costs known, pick the point farthest above the diagonal, where TPR − FPR is largest.

“AUC 0.95 means the model is great for my rare-fraud problem.” Maybe not. When positives are very rare, FPR divides by an enormous number of negatives, so thousands of false alarms still look like a tiny FPR and the ROC curve looks wonderful. Look at precision too, or at a precision-recall curve, which shows the false-alarm problem plainly.

Maths and principle

Principle: AUC equals the share of (positive, negative) pairs that the model puts in the right order.

Equation: with n1 positives scored si and n0 negatives scored sj,

AUC=1n1n0∑i∈pos∑j∈neg([si>sj]+12[si=sj])

The square brackets mean “1 if true, 0 if not”. This is the same number as the area under the ROC curve, and the Mann-Whitney U statistic divided by n1n0.

Try it: positives scored 0.9 and 0.6; negatives scored 0.7 and 0.2. AUC? (Pairs: 0.9 > 0.7 ✓, 0.9 > 0.2 ✓, 0.6 > 0.7 ✗, 0.6 > 0.2 ✓. AUC = 3/4 = 0.75.)

5. Worked example

Made-up numbers. Six patients get risk scores; three really have the condition (P), three don’t (N):

score 0.95 0.80 0.70 0.55 0.40 0.10
truth P P N P N N

Lower the threshold step by step, one patient at a time:

flag scores ≥ TP FP TPR FPR
1.00 0 0 0 0
0.95 1 0 0.33 0
0.80 2 0 0.67 0
0.70 2 1 0.67 0.33
0.55 3 1 1 0.33
0.40 3 2 1 0.67
0.10 3 3 1 1

The area under these steps: 0.67 × 0.33 (from FPR 0 to 0.33) + 1 × 0.67 (from 0.33 to 1) = 0.22 + 0.67 = 0.89. Check by pairs: of the 9 (P, N) pairs only one is in the wrong order (the P at 0.55 below the N at 0.70), so AUC = 8/9 ≈ 0.89. The threshold 0.55 catches every case with one false alarm, the best Youden point (TPR − FPR = 0.67).

6. Practice

  1. Model A has AUC 0.92, model B 0.78. Which ranks better?
  2. Why does AUC not depend on the threshold?
  3. If you flip a model’s scores (use 1 − score), what happens to its AUC?
Answers
  1. A: it puts positives above negatives more often.
  2. It sums up every threshold at once; it only depends on the order of the scores.
  3. It becomes 1 − AUC. A model with AUC 0.2 is a good model used upside down.
  • The ROC curve plots true positive rate against false positive rate for every threshold.
  • Random guessing gives the diagonal (AUC 0.5); a perfect model gives AUC 1.
  • AUC is the chance a random positive scores above a random negative: a measure of ranking.
  • Pick a working threshold from the curve using costs, or Youden’s J if costs are unknown.
  • For very rare positives, check precision too: a high AUC can hide many false alarms.

Try it on your own data: the ROC curve tool draws the ROC curve and AUC from your own labels and scores.

New words: ROC curve, true positive rate, false positive rate, AUC, Youden’s J, precision-recall curve.

Check yourself

1What does an AUC of 0.5 mean?
2AUC = 0.85 means:
3The positive class is only 0.1 % of cases. What can mislead?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.