Module 4: Classification and logistic regression · Lesson 4.1 · about 15 min
Predicting a category
Classification predicts which group something belongs to: spam or not, which digit, will this customer leave. See how it differs from regression, why models output probabilities, and why a straight line is the wrong tool for yes/no labels.
Module 4 · about 15 minutes · read first: 1.2. You’ll be able to tell classification from regression, name binary and multi-class problems, and explain why classifiers give probabilities.
1. Why this matters
Many of the most useful predictions are not numbers but categories (groups). Is this payment fraud? Is this X-ray normal? Will this machine break this week? Which shelf does this product go on? Predicting a category is called classification. Predicting a number, like a price, is regression (Module 3). This module shows the main ways to classify, starting from the straight-line model you already know.
2. Classification in one picture
Think of sorting post. A regression model would guess how heavy each letter is. A classifier puts each letter in a tray.
- Binary classification has two trays, usually coded 1 (“positive”: spam, fraud, disease) and 0 (“negative”). “Positive” only means “the thing you are looking for”. It is not good news.
- Multi-class has more than two trays: the digits 0 to 9, types of plant, which department a complaint goes to.
- Multi-label lets one item go in several trays at once: one photo can contain a dog and a car.
3. Probabilities, then decisions
A good classifier doesn’t just say “spam”. It says “spam, with probability 0.93”. A probability is a number from 0 (surely not) to 1 (surely yes). You then turn it into an action with a threshold, a cut-off line: flag the email if , or only if when a false alarm is costly.
Keep the two steps apart. The model’s job is to say how likely something is. Your job (or the business’s) is to decide how much risk is worth acting on. For multi-class problems the model gives one probability per class, they add up to 1, and it usually picks the biggest.
The model scores five card payments. Its fraud probabilities are 0.02, 0.15, 0.48, 0.81 and 0.97.
One threshold, 0.5. Payments 4 and 5 are blocked; the other three go through. Payment 3, at 0.48, slips past even though it is nearly a coin flip.
Think about costs. Blocking a real customer annoys them. Missing fraud loses money. A quick "was this you?" text is cheap.
Two thresholds, three actions. Above 0.9: block (payment 5). From 0.4 to 0.9: text the customer (payments 3 and 4). Below 0.4: let it through (payments 1 and 2).
Same model, better decisions. Nothing about the model changed. Only the cut-off lines moved, chosen from what each mistake costs.
- p(x)
- The model's probability that item x is in the positive class, written P(y = 1 | x): "the chance y is 1, given what we know about x".
- t
- The threshold: the cut-off. If p is at least t, predict 1; otherwise predict 0.
- ŷ
- "y-hat", the predicted label (the hat means "estimated").
- Physical meaning
- p is like a weather forecast of "70 % chance of rain". The threshold is your personal rule for carrying an umbrella, and it depends on how much you mind getting wet.
The best threshold comes from costs. If acting costs a little and missing costs a lot, act even when the chance is small.
4. Why not just use linear regression?
Code the labels as 0 and 1 and fit a straight line through them. It sort of works, but:
- A line has no ends. Its predictions run below 0 and above 1, and a “probability” of 1.3 makes no sense.
- Examples far from the border are the easiest to classify, yet they pull the line around the most, moving the border the wrong way.
- Squared error punishes a confident, correct 1.4 just as much as a wrong 0.6.
Logistic regression (Lessons 4.2 to 4.4) keeps the straight-line score but bends it through an S-shaped curve into a proper probability, and it uses a loss made for probabilities.
“A classifier that says 0.7 is right 70 % of the time.” Only if it is calibrated. Many models give scores that rank cases well but are too extreme or too timid as probabilities. Check this by grouping cases the model scored about 0.7 and seeing whether about 70 % of them really were positive. If not, the scores still sort cases well, but don’t read them as true chances.
Maths and principle
Principle: a classifier estimates the probability of each class given the inputs, then a decision rule turns it into a label.
Equations: for binary classification,
with threshold (often 0.5). For classes, with , and (“pick the class with the largest probability”).
Threshold from costs: if missing a positive costs and acting costs , acting pays when , so .
Analogy: a weather forecast. “70 % chance of rain” is the model; whether you carry an umbrella is the decision.
Try it: a model gives three classes probabilities 0.2, 0.5 and 0.3. Which class does it pick, and how sure is it? (Class 2, with 50 %: not very sure.)
5. Worked example
Made-up numbers. A bank’s model scores five card payments for fraud: 0.02, 0.15, 0.48, 0.81, 0.97. With threshold 0.5, payments 4 and 5 are blocked. But blocking a real customer’s payment is annoying, and missing fraud costs money. So the bank uses two thresholds: above 0.9, block automatically (payment 5); between 0.4 and 0.9, send a text asking the customer to confirm (payments 3 and 4); below 0.4, let it through. One model, three actions, all from the same probabilities.
6. Practice
- Binary, multi-class or multi-label? (a) Tag a news article with every topic it covers. (b) Is this email spam? (c) Which of 5 languages is this text in?
- Why might a hospital use a threshold below 0.5 for a screening test?
- Fitting a straight line to 0/1 labels gives a prediction of 1.3. What’s wrong with that?
Answers
- (a) Multi-label. (b) Binary. (c) Multi-class.
- Missing a sick patient is much worse than a false alarm that leads to a second test, so they flag anyone with even a moderate probability.
- A probability can’t be more than 1. The straight line has no limits.
- Classification predicts a category; regression predicts a number.
- Binary has two classes, multi-class more than two, multi-label allows several at once.
- Good classifiers give a probability; a threshold turns it into an action.
- Choose the threshold from the cost of each kind of mistake, not by habit at 0.5.
- A straight line is a poor classifier: it gives impossible probabilities and is pulled about by easy, far-away points.
Try it on your own data: the small model trainer trains logistic regression, a decision tree or k-nearest neighbours on a CSV with a category column.
New words: classification, binary, multi-class, multi-label, positive class, threshold, decision rule, calibrated.
Check yourself
Sources
- James, Witten, Hastie and Tibshirani (2021), An Introduction to Statistical Learning, 2nd edition, Springer, Chapter 4 (Classification) https://www.statlearning.com/
- Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning, 2nd edition, Springer, Chapter 4 (Linear methods for classification)
- scikit-learn user guide: Tuning the decision threshold for class prediction https://scikit-learn.org/stable/modules/classification_threshold.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.