Module 3: Linear regression: the first model · Lesson 3.7 · about 25 min
Overfitting and underfitting
A model can be too simple to catch the pattern, or so flexible it learns the noise. Watch training and test error as a polynomial gets more flexible, find the sweet spot, and learn the five main cures for overfitting.
Module 3 · about 25 minutes · read first: 2.5, 3.6. You’ll be able to recognise underfitting and overfitting from training and test errors, explain why test error is U-shaped, and choose a cure.
1. Why this matters
Overfitting is the central problem of machine learning. A model is useful only for data it hasn’t seen, yet it learns only from data it has. Every model in the rest of this course has a “flexibility dial” (the degree of a polynomial, the depth of a tree, the number of trees, the size of a network) and every one can be turned too far. Learning to spot it, and to stop before it, is what separates a working model from an impressive-looking broken one.
2. Two ways to fail
- Underfitting: the model is too simple to follow the real pattern. A straight line through a curve. Training and test errors are both high.
- Overfitting: the model is so flexible it fits the noise, the random wiggles of this particular sample. Training error is very low; test error is high. The model has memorised instead of learned.
Think of a student preparing for an exam. One reads only the chapter titles (underfitting) and can’t answer anything. Another memorises last year’s paper word for word, typos included (overfitting), and is lost when the questions change. The one who learns the ideas does best on the real exam.
Between them is a sweet spot where the test error is lowest.
The two curves on the right are the most important chart in machine learning. Training error falls for ever as flexibility grows: more freedom always fits the training rows better. Test error falls at first (the model catches the real curve) and then rises (it starts chasing noise).
Here is the U-shape in numbers, for 14 noisy points fitted with polynomials of rising degree:
3. Why it happens
The training data is the true pattern plus noise. A flexible model can’t tell the two apart, and given enough freedom it fits both. The noise is different in every sample, so the noise-fitting parts are pure error on new data.
Overfitting gets worse with:
- More flexibility: more parameters, more features, deeper trees.
- Less data: 14 points can be threaded by a degree-13 polynomial exactly; 14,000 cannot.
- More noise in the labels.
4. Cures
- Get more data. The most reliable cure. Noise averages out; the pattern doesn’t.
- Use a simpler model, or turn its dial down (lower degree, shallower tree).
- Add a penalty for complexity: regularisation (Lesson 3.8).
- Stop early: when training by gradient descent, stop when the validation error starts rising (Lesson 9.5).
- Average many models: random forests (Lesson 6.3) average away much of the noise-fitting.
And for underfitting: a more flexible model, better features (Lesson 3.6), or less penalty.
“My model is 100 % right on the training data, so it’s excellent.” That is usually a warning sign, not a triumph. A flexible enough model can memorise any training set, noise and all. Only the score on rows the model has never seen tells you whether it learned something real. A big gap between training and test scores is the signature of overfitting.
Maths and principle
Principle: the expected error on new data splits into three parts.
Equation: for squared error, at a given input,
- Bias is how far the average fitted model (over many possible training sets) is from the truth. Simple models have high bias.
- Variance is how much the fitted model changes from one training set to another. Flexible models have high variance.
- Noise can’t be removed by any model.
Turning up flexibility lowers bias and raises variance; the test error is lowest where their sum is smallest. Lesson 7.1 shows this with a dartboard.
Analogy: a tailor. Off-the-peg sizes (underfitting) fit nobody well. A suit cut to every crease of the clothes you wore that day (overfitting) won’t fit tomorrow. A good tailor fits the body, not the creases.
Try it: with 10 training points, what degree of polynomial can pass through all of them exactly? (Degree 9: 10 weights for 10 points. Its training error is 0, and almost certainly it overfits.)
- Bias²
- How far the model's average guess is from the truth: the error from being too simple.
- Variance
- How much the fitted model jumps around when trained on a different sample: the error from being too jumpy.
- σ² (noise)
- Randomness in the labels themselves, which no model can predict.
- Sweet spot
- The flexibility where bias² + variance is smallest, so the test error is lowest.
- Physical meaning
- A tailor. Off-the-peg sizes (high bias) fit nobody well. A suit cut to every crease of the clothes you happened to wear today (high variance) won't fit tomorrow. A good tailor fits the body, not the creases, and the creases are the noise.
5. Worked example
Made-up numbers. A team predicts monthly energy use of 300 office buildings from 40 features. Results:
| Model | Training RMSE | Validation RMSE |
|---|---|---|
| Mean of all buildings (baseline) | 48 | 49 |
| Linear regression, 5 features | 31 | 32 |
| Linear regression, all 40 features | 24 | 29 |
| Degree-2 polynomial, all features (861 columns) | 3 | 71 |
The 5-feature model underfits a little (training and validation close but high). The 40-feature model is the best on validation. The polynomial model, with more columns than buildings, memorises: training error nearly zero, validation error worse than the baseline. The team keeps the 40-feature model and tries a penalty to see if it can do better still.
6. Practice
- A tree of depth 20 has training accuracy 100 % and test accuracy 71 %; depth 5 gives 86 % and 84 %. Which do you choose, and why?
- You can’t collect more data. Name two other cures for overfitting.
- Why does training error keep falling as flexibility grows, even past the sweet spot?
Answers
- Depth 5: its test accuracy is higher, and its small train-test gap shows it hasn’t memorised.
- A simpler model, a penalty (regularisation), early stopping, or averaging many models.
- A more flexible model can do everything a less flexible one can, and more, so on the rows it is fitted to it can only do as well or better.
Try it on your own data: the small model trainer shows training and test scores side by side as you change a tree’s depth or k in k-nearest neighbours.
New words: underfitting, overfitting, sweet spot, noise, bias and variance, early stopping.
- Underfitting: too simple, both errors high. Overfitting: too flexible, training error low but test error high.
- Training error always falls as flexibility grows; test error falls then rises in a U.
- Expected test error = bias² + variance + noise; flexibility trades bias for variance.
- Cures for overfitting: more data, a simpler model, a penalty, early stopping, or averaging many models.
Check yourself
Sources
- James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Sections 2.2.1 to 2.2.2 (the bias-variance trade-off) https://www.statlearning.com/
- Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd ed., Springer, Chapter 7 (model assessment and selection) https://hastie.su.domains/ElemStatLearn/
- Geman, S., Bienenstock, E. and Doursat, R. (1992), Neural Networks and the Bias/Variance Dilemma, Neural Computation 4(1), 1-58
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.