Module 4: Classification and logistic regression · Lesson 4.3 · about 30 min
Log-loss: how logistic regression learns
Logistic regression is trained by making the log-loss (cross-entropy) small. See why it punishes confident wrong answers so hard, derive its surprisingly simple gradient, and connect it to maximum likelihood.
Module 4 · about 30 minutes · read first: 1.5, 3.3, 4.2. You’ll be able to compute log-loss, explain why it suits probabilities, derive the gradient of logistic regression, and say what maximum likelihood means.
1. Why this matters
A model that gives probabilities needs a score that rewards good probabilities, not just right yes/no answers. Log-loss, also called binary cross-entropy, is that score. Its many-class version, cross-entropy, is exactly what language models make small when they learn to predict the next word (Lesson 10.6). Learn it here on one neuron and you know the training rule of an LLM.
2. The loss for one example
Think of it as a “surprise bill”. If the truth is “yes” (), the bill is . If the truth is “no” (), it is . Here ln is the natural logarithm and the model’s probability of “yes”. One formula covers both cases, because one of the two terms is always zero:
- Right and sure ( for a yes): bill 0.01.
- Unsure (): bill 0.69 either way.
- Wrong and sure ( for a yes): bill 4.6, and it heads to infinity as heads to 0.
The average bill over all training examples is the model’s log-loss. Training finds the weights that make it as small as possible.
- y
- The true label: 1 for yes, 0 for no.
- p
- The model's probability of yes.
- −ln p
- The surprise when a yes happens: 0 if p was 1, larger the lower p was.
- ℓ
- The loss for one example; the log-loss is its average over all examples.
- Physical meaning
- It is how a fair forecaster is scored over a year. Saying "99 % sun" on a day it pours costs far more than saying "60 %". The best long-run score comes from stating what you truly believe, so the model can't game it.
3. The gradient: error times input
To walk downhill we need the slope (gradient) of the loss for each weight. With and , use the chain rule in three links: loss → → → .
- (the sigmoid’s slope from Lesson 4.2)
Multiply them and the cancels:
Averaged over the training set, that is the gradient. It has exactly the same shape as linear regression’s (Lesson 3.3): error times input, with the error now a gap in probability. There is no formula that jumps straight to the best weights this time, so we walk downhill with gradient descent. Luckily the log-loss is a perfect bowl (convex), so there is only one bottom to find.
Start flat. Three students studied 1, 2 and 3 hours; only the last passed (y = 0, 0, 1). With w = 0 and b = 0, every p = σ(0) = 0.5. Log-loss 0.693.
Errors. p − y = 0.5, 0.5, −0.5. Two predictions were too high, one too low.
Gradient for w. Average of error × hours: (0.5 × 1 + 0.5 × 2 − 0.5 × 3) ÷ 3 = 0. The hours cancel out for now, so w stays put.
Gradient for b. Average error: 0.5 ÷ 3 = 0.167. Step against it (learning rate 1): b = −0.167. The model first learns "most students fail".
Next step. Now every p = 0.458, errors 0.458, 0.458, −0.542. The w gradient is (0.458 + 0.917 − 1.625) ÷ 3 = −0.083, so w starts to rise: more hours, more passing.
“A model with 90 % accuracy must have a low log-loss.” Not necessarily. Accuracy only checks which side of 0.5 each prediction falls. Log-loss also checks how sure the model was. A model that is right 90 % of the time but says 0.999 on its wrong answers can have a worse log-loss than a humbler model with lower accuracy.
Maths and principle
Principle: making log-loss small is the same as maximum likelihood: choosing the weights that make the labels we actually saw as probable as possible.
Equations: the probability the model gives to the whole training set (treating examples as independent) is the likelihood
Each factor is for a yes and for a no. Products of many small numbers are awkward, so take logs (which turn products into sums) and a minus sign (to minimise instead of maximise):
which is the mean log-loss. Same best weights either way.
Analogy: a weather forecaster scored over a year, as in the box above.
Try it: two examples: a yes with and a no with . Mean log-loss? (.)
4. Worked example
Made-up numbers. One input, three students: 1, 2, 3 hours, 0, 0, 1. Start at , , learning rate 1.
- All . Errors : 0.5, 0.5, −0.5. Log-loss = 0.693.
- Gradient for : . For : .
- Update: , . The model first learns “most students fail” (only 1 of 3 passed) by lowering the bias.
- Next step, for all. Errors 0.458, 0.458, −0.542. Gradient for : : now starts to rise, because the one pass had the most hours.
After a few hundred steps the model settles with a steep positive . (With data this tidy, where one cut-off separates the classes perfectly, would grow for ever: the loss keeps falling as the S gets steeper. A small L2 penalty, Lesson 3.8, stops that.)
5. Practice
- A yes is predicted at . What is its log-loss?
- Which costs more: predicting 0.99 for a no, or 0.6 for a no?
- What is for a yes predicted at 0.7?
Answers
- .
- 0.99 for a no: , against .
- : the bias should go up.
- Log-loss for one example is −ln p if the truth is yes and −ln(1 − p) if it is no.
- It punishes confident wrong answers very hard, and rewards honest probabilities.
- Its gradient is simply (p − y) × x: error times input.
- The log-loss of logistic regression is a convex bowl, so gradient descent finds the single best answer.
- Making log-loss small is the same as maximum likelihood.
Try it on your own data: the small model trainer trains logistic regression on your CSV by exactly this gradient descent, and shows the test accuracy.
New words: log-loss, binary cross-entropy, likelihood, maximum likelihood, convex.
Check yourself
Sources
- Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, Section 5.5 (Maximum likelihood estimation) and 6.2.2 https://www.deeplearningbook.org/
- Bishop (2006), Pattern Recognition and Machine Learning, Springer, Section 4.3.2 (cross-entropy error for logistic regression)
- scikit-learn API reference: sklearn.metrics.log_loss https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.