Home / Learn / Data & AI / ML foundations / Lesson 3.7
All lessons

Module 3: Linear regression: the first model · Lesson 3.7 · about 25 min

Overfitting and underfitting

A model can be too simple to catch the pattern, or so flexible it learns the noise. Watch training and test error as a polynomial gets more flexible, find the sweet spot, and learn the five main cures for overfitting.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 25 minutes · read first: 2.5, 3.6. You’ll be able to recognise underfitting and overfitting from training and test errors, explain why test error is U-shaped, and choose a cure.

A model that fits its training data too closely learns the noise as well as the pattern, and does worse on new data: the art is to stop at the sweet spot.

1. Why this matters

Overfitting is the central problem of machine learning. A model is useful only for data it hasn’t seen, yet it learns only from data it has. Every model in the rest of this course has a “flexibility dial” (the degree of a polynomial, the depth of a tree, the number of trees, the size of a network) and every one can be turned too far. Learning to spot it, and to stop before it, is what separates a working model from an impressive-looking broken one.

2. Two ways to fail

  • Underfitting: the model is too simple to follow the real pattern. A straight line through a curve. Training and test errors are both high.
  • Overfitting: the model is so flexible it fits the noise, the random wiggles of this particular sample. Training error is very low; test error is high. The model has memorised instead of learned.

Think of a student preparing for an exam. One reads only the chapter titles (underfitting) and can’t answer anything. Another memorises last year’s paper word for word, typos included (overfitting), and is lost when the questions change. The one who learns the ideas does best on the real exam.

Between them is a sweet spot where the test error is lowest.

Try it: find the sweet spot
training error (RMSE)
test error (RMSE)
parameters learned

Left: circles are the 14 training points; small squares are 60 test points the fit never sees. Right: error against degree. The training error only ever falls as the model gets more flexible, but the test error falls and then climbs again: the model has started to fit the noise. The best degree sits at the bottom of the test curve.

The two curves on the right are the most important chart in machine learning. Training error falls for ever as flexibility grows: more freedom always fits the training rows better. Test error falls at first (the model catches the real curve) and then rises (it starts chasing noise).

Here is the U-shape in numbers, for 14 noisy points fitted with polynomials of rising degree:

Degree 1 (a straight line). Training error 9.0, test error 9.5. Both high and close together: the line is too stiff for the curve. Underfitting.
Degree 3. Training error 2.1, test error 2.6. Both low: the model follows the real curve. The sweet spot.
Degree 8. Training error 1.2, test error 4.8. Training still improves, but test error is climbing: the extra bends are fitting noise.
Degree 13. Training error 0: the curve threads all 14 points. Test error 40: wild swings between them. The model has memorised, not learned.
The rule. Choose the flexibility with the lowest validation error, never the lowest training error. (The numbers are illustrative; the simulation above shows the same shape.)

3. Why it happens

The training data is the true pattern plus noise. A flexible model can’t tell the two apart, and given enough freedom it fits both. The noise is different in every sample, so the noise-fitting parts are pure error on new data.

Overfitting gets worse with:

  • More flexibility: more parameters, more features, deeper trees.
  • Less data: 14 points can be threaded by a degree-13 polynomial exactly; 14,000 cannot.
  • More noise in the labels.

4. Cures

  1. Get more data. The most reliable cure. Noise averages out; the pattern doesn’t.
  2. Use a simpler model, or turn its dial down (lower degree, shallower tree).
  3. Add a penalty for complexity: regularisation (Lesson 3.8).
  4. Stop early: when training by gradient descent, stop when the validation error starts rising (Lesson 9.5).
  5. Average many models: random forests (Lesson 6.3) average away much of the noise-fitting.

And for underfitting: a more flexible model, better features (Lesson 3.6), or less penalty.

“My model is 100 % right on the training data, so it’s excellent.” That is usually a warning sign, not a triumph. A flexible enough model can memorise any training set, noise and all. Only the score on rows the model has never seen tells you whether it learned something real. A big gap between training and test scores is the signature of overfitting.

Maths and principle

Principle: the expected error on new data splits into three parts.

Equation: for squared error, at a given input,

𝔼[(y−yˆ)2]=(bias)2⏟too simple+variance⏟too jumpy+σ2⏟noise
  • Bias is how far the average fitted model (over many possible training sets) is from the truth. Simple models have high bias.
  • Variance is how much the fitted model changes from one training set to another. Flexible models have high variance.
  • Noise σ2 can’t be removed by any model.

Turning up flexibility lowers bias and raises variance; the test error is lowest where their sum is smallest. Lesson 7.1 shows this with a dartboard.

Analogy: a tailor. Off-the-peg sizes (underfitting) fit nobody well. A suit cut to every crease of the clothes you wore that day (overfitting) won’t fit tomorrow. A good tailor fits the body, not the creases.

Try it: with 10 training points, what degree of polynomial can pass through all of them exactly? (Degree 9: 10 weights for 10 points. Its training error is 0, and almost certainly it overfits.)

Bias²
How far the model's average guess is from the truth: the error from being too simple.
Variance
How much the fitted model jumps around when trained on a different sample: the error from being too jumpy.
σ² (noise)
Randomness in the labels themselves, which no model can predict.
Sweet spot
The flexibility where bias² + variance is smallest, so the test error is lowest.
Physical meaning
A tailor. Off-the-peg sizes (high bias) fit nobody well. A suit cut to every crease of the clothes you happened to wear today (high variance) won't fit tomorrow. A good tailor fits the body, not the creases, and the creases are the noise.

5. Worked example

Made-up numbers. A team predicts monthly energy use of 300 office buildings from 40 features. Results:

Model Training RMSE Validation RMSE
Mean of all buildings (baseline) 48 49
Linear regression, 5 features 31 32
Linear regression, all 40 features 24 29
Degree-2 polynomial, all features (861 columns) 3 71

The 5-feature model underfits a little (training and validation close but high). The 40-feature model is the best on validation. The polynomial model, with more columns than buildings, memorises: training error nearly zero, validation error worse than the baseline. The team keeps the 40-feature model and tries a penalty to see if it can do better still.

6. Practice

  1. A tree of depth 20 has training accuracy 100 % and test accuracy 71 %; depth 5 gives 86 % and 84 %. Which do you choose, and why?
  2. You can’t collect more data. Name two other cures for overfitting.
  3. Why does training error keep falling as flexibility grows, even past the sweet spot?
Answers
  1. Depth 5: its test accuracy is higher, and its small train-test gap shows it hasn’t memorised.
  2. A simpler model, a penalty (regularisation), early stopping, or averaging many models.
  3. A more flexible model can do everything a less flexible one can, and more, so on the rows it is fitted to it can only do as well or better.

Try it on your own data: the small model trainer shows training and test scores side by side as you change a tree’s depth or k in k-nearest neighbours.

New words: underfitting, overfitting, sweet spot, noise, bias and variance, early stopping.

  • Underfitting: too simple, both errors high. Overfitting: too flexible, training error low but test error high.
  • Training error always falls as flexibility grows; test error falls then rises in a U.
  • Expected test error = bias² + variance + noise; flexibility trades bias for variance.
  • Cures for overfitting: more data, a simpler model, a penalty, early stopping, or averaging many models.

Check yourself

1Training error 2 %, test error 25 %. What is most likely?
2Training error 30 %, test error 31 %, and a simple baseline gets 32 %. What is most likely?
3Which of these does NOT help against overfitting?

Sources

  • James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Sections 2.2.1 to 2.2.2 (the bias-variance trade-off) https://www.statlearning.com/
  • Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd ed., Springer, Chapter 7 (model assessment and selection) https://hastie.su.domains/ElemStatLearn/
  • Geman, S., Bienenstock, E. and Doursat, R. (1992), Neural Networks and the Bias/Variance Dilemma, Neural Computation 4(1), 1-58

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.