Home / Learn / Data & AI / ML foundations / Lesson 7.1
All lessons

Module 7: Judging models honestly · Lesson 7.1 · about 25 min

Bias and variance

Every model's test error splits into bias (wrong on average), variance (changes with the sample) and noise (nobody can remove). Train the same model on 20 different samples and see both, then read the dartboard.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 7 · about 25 minutes · read first: 3.7, 6.1. You’ll be able to name the three parts of test error, tell from training and validation scores whether a model is too simple or too jumpy, and pick the right fix for each.

A model can be wrong in two ways: it misses the same way every time (bias), or it changes a lot with each new sample (variance), and the cures are opposite.

1. Why this matters

When a model disappoints, ask first: is it too simple, or too jumpy? The two problems need opposite fixes. One needs more flexibility, the other needs less. Guess wrong and you can waste days. Bias and variance are the two words for this question. They also explain why random forests average many trees and why boosting adds trees up (Module 6).

2. Two ways to be wrong

Picture an archer. Some archers always hit low and to the left: their arrows land close together, but in the wrong place. Others aim at the centre on average, but their arrows fly all over the board. Models behave the same way.

Imagine training the same kind of model many times, each time on a fresh sample from the same source.

  • Bias is how far the average of those models is from the truth. A model that is too simple misses the same way every time, like the archer who always hits low.
  • Variance is how much the models differ from each other. A model that is too flexible follows the noise in each sample, so every sample gives a different fit, like the archer whose arrows scatter.
Picture: the dartboard Four dartboards. Low bias and low variance: darts tight on the bullseye. Low bias, high variance: darts scattered all round the bullseye. High bias, low variance: a tight cluster off to one side. High bias, high variance: a wide cluster off to one side. low bias, low variance the goal low bias, high variance overfitting high bias, low variance underfitting high bias, high variance the worst fibub.com
Each cross is one model, trained on a different sample of data; the bullseye is the true answer. Bias: the crosses' centre misses the bullseye, the same way every time. Variance: the crosses scatter, because each model depends heavily on its own sample.

The simulation does this for real. It takes 20 samples of 12 points from one curve and fits 20 polynomials, so you can see how far the fits wander and where their average lands.

Try it: 20 training sets, 20 models
bias² (average miss, squared)
variance (spread of fits)
bias² + variance + noise
Each faint curve is the same kind of model trained on a different random sample of 12 points from the dashed curve. Degree 0 or 1 is too stiff: every fit misses in the same way (high bias, low variance). Degree 8 or 9 is too wobbly: each fit chases its own sample (low bias, high variance). The expected test error is bias² plus variance plus the noise nobody can remove (here 0.09); it is lowest in between, around degree 3.

Watch it happen

Here is the whole idea with five small numbers. At one point the true value is 5.

Train five times. The same model is trained on five different samples. At our point it predicts 6, 7, 6, 5 and 6. The truth is 5.

Find the centre. The average prediction is (6 + 7 + 6 + 5 + 6) ÷ 5 = 6. The models miss by 1 on average, all in the same direction.

Bias². Square that miss: (6 − 5)² = 1. This is the part of the error that more data will not fix, because the model itself is off.

Variance. How far does each prediction sit from the centre of 6? Distances 0, 1, 0, −1, 0. Squares 0, 1, 0, 1, 0. Average: 2 ÷ 5 = 0.4.

Add the noise. Real data also wobble by themselves. Say that noise has variance 0.6. Expected squared error = 1 + 0.4 + 0.6 = 2.0. Each part has a different cure.

3. The trade-off and the cures

More flexibility lowers bias and raises variance. Less flexibility does the opposite. Test error is lowest somewhere in between: the U-shaped curve of Lesson 3.7.

You can’t watch bias and variance directly on real data, because you only have one sample. But training and validation error give the game away:

training error validation error diagnosis fixes
A high high, close to training high bias (underfitting) more flexible model, better features, less regularisation, boosting
B low much higher high variance (overfitting) more data, simpler model, regularisation, averaging (bagging, forests), early stopping
C low low, close good ship it

The third part, noise, is the randomness in the data itself: measurement error, or things your columns don’t capture. No model can remove it. Only better data can.

y = f(x) + ε
what we observe: a true pattern f plus random noise ε
f̂ (f-hat)
the model you get from one particular training sample
Bias²
how far the average model is from the truth, squared: the error built into the kind of model you chose
Variance
how much the model changes from one sample to the next: the error that comes from the luck of the sample
σ² (noise)
the wobble in the data that no model can predict
Physical meaning
A forecaster that always says "same as last week" is steady but stubbornly wrong (bias). One that rebuilds itself from scratch every Monday is clever but nervous (variance). Good models sit between, and part of tomorrow's sales will always be a surprise (noise).
Maths and principle

Principle: averaged over possible training sets, squared test error splits into three parts that can be measured separately.

The decomposition: data come from y=f(x)+ε with noise variance σ2. Let fˆ be the model trained on a random sample, and f‾(x)=E[fˆ(x)] its average over samples (E means “average over many tries”). At a point x:

E[(y−fˆ(x))2]=bias2+variance+σ2

where

bias2=(f‾(x)−f(x))2
variance=E[(fˆ(x)−f‾(x))2]

The cross terms vanish because the noise in a new y is independent of the training sample.

Analogy: the dartboard. Bias is where the cluster of darts is centred; variance is how widely they scatter.

Try it: at a point, the truth is 5. Five models trained on different samples predict 6, 7, 6, 5, 6. Bias² and variance? (Average 6, so bias² = 1. Variance = (0 + 1 + 0 + 1 + 0)/5 = 0.4.)

“More data always helps.” More data mostly cuts variance: each fit depends less on the luck of the sample. It does little for bias. A straight line fitted to a million points from a curve is still a straight line, missing the curve in the same places. If training and validation error are both high and close, collect better features or use a more flexible model, not just more rows.

4. Worked example

Made-up numbers. A shop forecasts daily sales. Errors are root mean squared (RMSE), in units sold.

model training RMSE validation RMSE
average of last week 41 42
linear model on 6 columns 30 31
boosted trees, 2,000 rounds, depth 10 4 38
boosted trees, early stopping (310 rounds), depth 4 21 24

The first two are close on training and validation: high bias. The third has a huge gap: high variance. The fourth is the best compromise. The noise floor is probably near 20: even perfect knowledge of the columns can’t predict a sudden bus strike.

5. Practice

  1. In the simulation, which degree gives the lowest bias² + variance?
  2. Training error 3 %, validation error 3.5 %, but the business needs 1 %. Bias or variance problem?
  3. Why does a random forest mostly reduce variance, and boosting mostly bias?
Answers
  1. Around 3 on most draws (degree 1 is a close second, because this curve is gentle).
  2. Bias: training and validation agree, so the model isn’t overfitting; it needs more flexibility or better features.
  3. A forest averages deep trees that already have low bias, cancelling their variance. Boosting adds up very simple, high-bias pieces until the sum is flexible enough.

Try it on your own data: the small model trainer shows training and test accuracy side by side; use the table above to diagnose your model.

New words: bias, variance, irreducible error (noise), bias-variance decomposition, learning curve.

  • Test error = bias² + variance + noise.
  • Bias: the model misses the same way every time (too simple). Variance: the model changes a lot with the sample (too flexible).
  • High training and validation error, close together, means high bias. Low training error with much higher validation error means high variance.
  • More data, simpler models and averaging cut variance; more flexibility and better features cut bias. Nothing removes noise except better data.

Check yourself

1A straight line fitted to a strongly curved pattern has
2Which change mainly lowers variance?
3Expected test error (squared) equals:

Sources

  • James, Witten, Hastie, Tibshirani and Taylor (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Section 2.2.2 (the bias-variance trade-off) https://www.statlearning.com/
  • Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning, 2nd edition, Springer, Sections 7.2 and 7.3
  • Geman, Bienenstock and Doursat (1992), Neural networks and the bias/variance dilemma, Neural Computation 4(1), 1-58 https://doi.org/10.1162/neco.1992.4.1.1

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.