Home / Learn / Data & AI / ML foundations / Lesson 4.3
All lessons

Module 4: Classification and logistic regression · Lesson 4.3 · about 30 min

Log-loss: how logistic regression learns

Logistic regression is trained by making the log-loss (cross-entropy) small. See why it punishes confident wrong answers so hard, derive its surprisingly simple gradient, and connect it to maximum likelihood.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 4 · about 30 minutes · read first: 1.5, 3.3, 4.2. You’ll be able to compute log-loss, explain why it suits probabilities, derive the gradient of logistic regression, and say what maximum likelihood means.

Log-loss charges a model for how surprised it was by the truth, so a confident wrong answer costs far more than a humble one.

1. Why this matters

A model that gives probabilities needs a score that rewards good probabilities, not just right yes/no answers. Log-loss, also called binary cross-entropy, is that score. Its many-class version, cross-entropy, is exactly what language models make small when they learn to predict the next word (Lesson 10.6). Learn it here on one neuron and you know the training rule of an LLM.

2. The loss for one example

Think of it as a “surprise bill”. If the truth is “yes” (y=1), the bill is −ln⁡p. If the truth is “no” (y=0), it is −ln⁡(1−p). Here ln is the natural logarithm and p the model’s probability of “yes”. One formula covers both cases, because one of the two terms is always zero:

ℓ(y,p)=−[yln⁡p+(1−y)ln⁡(1−p)]
Try it: how much does a wrong guess cost?
loss if the truth is yes: −ln p
loss if the truth is no: −ln(1 − p)
squared error if yes: (1 − p)²
Log-loss charges almost nothing for a confident right answer and an ever-growing amount for a confident wrong one: at p = 0.01 for something that turns out true, the loss is 4.6, against 0.98 for squared error. That steep wall is what pushes logistic regression to get its probabilities right, not just its yes/no.
  • Right and sure (p=0.99 for a yes): bill 0.01.
  • Unsure (p=0.5): bill 0.69 either way.
  • Wrong and sure (p=0.01 for a yes): bill 4.6, and it heads to infinity as p heads to 0.

The average bill over all training examples is the model’s log-loss. Training finds the weights that make it as small as possible.

y
The true label: 1 for yes, 0 for no.
p
The model's probability of yes.
−ln p
The surprise when a yes happens: 0 if p was 1, larger the lower p was.
ℓ
The loss for one example; the log-loss is its average over all examples.
Physical meaning
It is how a fair forecaster is scored over a year. Saying "99 % sun" on a day it pours costs far more than saying "60 %". The best long-run score comes from stating what you truly believe, so the model can't game it.

3. The gradient: error times input

To walk downhill we need the slope (gradient) of the loss for each weight. With pi=σ(zi) and zi=𝐰⋅𝐱i+b, use the chain rule in three links: loss → p → z → wj.

  • ∂ℓ∂p=−yp+1−y1−p=p−yp(1−p)
  • ∂p∂z=p(1−p) (the sigmoid’s slope from Lesson 4.2)
  • ∂z∂wj=xj

Multiply them and the p(1−p) cancels:

∂ℓ∂wj=(p−y)xj,∂ℓ∂b=p−y

Averaged over the training set, that is the gradient. It has exactly the same shape as linear regression’s (Lesson 3.3): error times input, with the error now a gap in probability. There is no formula that jumps straight to the best weights this time, so we walk downhill with gradient descent. Luckily the log-loss is a perfect bowl (convex), so there is only one bottom to find.

Start flat. Three students studied 1, 2 and 3 hours; only the last passed (y = 0, 0, 1). With w = 0 and b = 0, every p = σ(0) = 0.5. Log-loss 0.693.

Errors. p − y = 0.5, 0.5, −0.5. Two predictions were too high, one too low.

Gradient for w. Average of error × hours: (0.5 × 1 + 0.5 × 2 − 0.5 × 3) ÷ 3 = 0. The hours cancel out for now, so w stays put.

Gradient for b. Average error: 0.5 ÷ 3 = 0.167. Step against it (learning rate 1): b = −0.167. The model first learns "most students fail".

Next step. Now every p = 0.458, errors 0.458, 0.458, −0.542. The w gradient is (0.458 + 0.917 − 1.625) ÷ 3 = −0.083, so w starts to rise: more hours, more passing.

“A model with 90 % accuracy must have a low log-loss.” Not necessarily. Accuracy only checks which side of 0.5 each prediction falls. Log-loss also checks how sure the model was. A model that is right 90 % of the time but says 0.999 on its wrong answers can have a worse log-loss than a humbler model with lower accuracy.

Maths and principle

Principle: making log-loss small is the same as maximum likelihood: choosing the weights that make the labels we actually saw as probable as possible.

Equations: the probability the model gives to the whole training set (treating examples as independent) is the likelihood

ℒ(𝐰)=∏i=1npiyi(1−pi)1−yi

Each factor is pi for a yes and 1−pi for a no. Products of many small numbers are awkward, so take logs (which turn products into sums) and a minus sign (to minimise instead of maximise):

−1nln⁡ℒ(𝐰)=−1n∑i=1n[yiln⁡pi+(1−yi)ln⁡(1−pi)]

which is the mean log-loss. Same best weights either way.

Analogy: a weather forecaster scored over a year, as in the box above.

Try it: two examples: a yes with p=0.8 and a no with p=0.3. Mean log-loss? ((−ln⁡0.8−ln⁡0.7)/2=(0.223+0.357)/2=0.29.)

4. Worked example

Made-up numbers. One input, three students: x= 1, 2, 3 hours, y= 0, 0, 1. Start at w=0, b=0, learning rate 1.

  • All p=σ(0)=0.5. Errors p−y: 0.5, 0.5, −0.5. Log-loss = 0.693.
  • Gradient for w: 13(0.5×1+0.5×2−0.5×3)=0. For b: 13(0.5)=0.167.
  • Update: w=0, b=−0.167. The model first learns “most students fail” (only 1 of 3 passed) by lowering the bias.
  • Next step, p=σ(−0.167)=0.458 for all. Errors 0.458, 0.458, −0.542. Gradient for w: 13(0.458+0.917−1.625)=−0.083: now w starts to rise, because the one pass had the most hours.

After a few hundred steps the model settles with a steep positive w. (With data this tidy, where one cut-off separates the classes perfectly, w would grow for ever: the loss keeps falling as the S gets steeper. A small L2 penalty, Lesson 3.8, stops that.)

5. Practice

  1. A yes is predicted at p=0.5. What is its log-loss?
  2. Which costs more: predicting 0.99 for a no, or 0.6 for a no?
  3. What is ∂ℓ/∂b for a yes predicted at 0.7?
Answers
  1. −ln⁡0.5≈0.693.
  2. 0.99 for a no: −ln⁡(0.01)≈4.6, against −ln⁡(0.4)≈0.92.
  3. p−y=0.7−1=−0.3: the bias should go up.
  • Log-loss for one example is −ln p if the truth is yes and −ln(1 − p) if it is no.
  • It punishes confident wrong answers very hard, and rewards honest probabilities.
  • Its gradient is simply (p − y) × x: error times input.
  • The log-loss of logistic regression is a convex bowl, so gradient descent finds the single best answer.
  • Making log-loss small is the same as maximum likelihood.

Try it on your own data: the small model trainer trains logistic regression on your CSV by exactly this gradient descent, and shows the test accuracy.

New words: log-loss, binary cross-entropy, likelihood, maximum likelihood, convex.

Check yourself

1The truth is "yes" and the model says p = 0.9. What is the log-loss for this example?
2The gradient of the log-loss for logistic regression with respect to a weight wⱼ is:
3Why not use squared error for logistic regression?

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.