Module 2: Looking at data before modelling · Lesson 2.5 · about 20 min
Train, validation and test sets
A model must be judged on data it never saw while learning. Split the data into training, validation and test sets, fit a line to the training points, and watch how the honest test score changes from split to split.
Module 2 · about 20 minutes · read first: 1.7. You’ll be able to split a dataset into training, validation and test sets, explain why each exists, and choose a split that fits the data (random, grouped or by time).
1. Why this matters
Imagine a student who memorises last year’s exam answers. On last year’s paper they score 100 %; on this year’s they may fail. A model can do the same: fit its training rows perfectly and be useless on new ones. The only honest check is to hide some data from the model while it learns, like a teacher keeping the real exam paper locked in a drawer, and then score it on that hidden data.
2. Three sets
| Set | Typical share | Used for | Touched |
|---|---|---|---|
| Training | 60 to 80 % | Fitting the parameters | Constantly |
| Validation | 10 to 20 % | Choosing the model type and its settings | Many times |
| Test | 10 to 20 % | One final, unbiased score | Once, at the end |
Why both validation and test? Every time you look at a score and change something because of it, a little information about that data leaks into your choices. After fifty tries on the validation set, its score is flattering. The test set, looked at once, isn’t.
With small datasets, setting aside a fixed validation set wastes rows. Cross-validation (Lesson 7.2) reuses each row for both training and validation in turns.
3. Try it
Things to notice:
- The training error is usually lower than the test error. The line was chosen to suit the training points.
- Each shuffle gives a different test error. With 10 test rows, a couple of unusual flats can swing the score. Bigger test sets are steadier, but leave less for training: a trade-off.
4. Choosing how to split
- Random split: fine when rows are independent (separate customers, separate homes).
- Stratified split: for classification with a rare class, keep the same share of each class in every set. Otherwise a test set of 50 rows might contain none of the 2 % fraud cases.
- Grouped split: when rows come in groups (several visits per patient, several photos per person), keep each group entirely in one set. Otherwise the model can recognise the patient instead of learning the disease.
- Time split: for anything over time, train on the past and test on the future. Never shuffle.
“I only peeked at the test set a few times, so its score is still fair.” Every peek that changes a decision leaks a little of the test set into the model. Tune against it fifty times and you have, in effect, trained on it: the score flatters you. Use the validation set (or cross-validation) for all choices, and look at the test set once.
Maths and principle
Principle: the test error estimates the generalisation error, the average error on new data from the same source. The training error underestimates it.
Equations: with a model fitted on the training set,
where is the loss for one row (for example the squared error ). The test error is itself an average of random values, so it has a standard error (Lesson 1.6) that shrinks like : four times the test rows halve its wobble.
Analogy: a driving test on roads the learner has never driven. Practising on the test route would make a pass meaningless.
Try it: a classifier gets 46 of 50 test rows right: 92 %. Roughly how uncertain is that? (The standard error of a proportion is , so about ±4 points, and a 95 % range of roughly 84 % to 100 %.)
- Training set
- The rows the model learns its parameters from.
- Validation set
- Rows used to compare models and settings, many times.
- Test set
- Rows kept locked away for one final score.
- Generalisation error
- How wrong the model will be, on average, on new data it has never seen. The test error estimates it.
- √(p(1 − p)/n)
- How much a test accuracy p, measured on n rows, could wobble just by luck of which rows were picked.
- Physical meaning
- A driving test on roads the learner has never driven. Practising on the test route would make a pass meaningless; the score only means something because the roads were new.
5. Worked example
Made-up numbers. A bank has 10,000 past loans, 300 of which were not repaid (3 %).
- Hold out a stratified 20 % test set: 2,000 loans with 60 defaults, kept the same 3 %.
- From the remaining 8,000, hold out 20 % for validation: 1,600 loans with 48 defaults.
- Train on the 6,400 left (192 defaults). Try logistic regression, a tree and a forest, with a few settings each, comparing them on validation.
- Pick the best, retrain it on training + validation (8,000 loans), and score it once on the test set.
- The loans were made over 5 years, so the bank also checks a time split: train on years 1 to 4, test on year 5. The score is a little lower, which is the more realistic number for next year.
6. Practice
- You tune a model 200 times against the test set and pick the best. Is its test score still honest?
- A dataset has 40 photos each of 50 people, for a model that detects glasses. How should you split?
- Why are training errors usually lower than test errors?
Answers
- No. The test set has become a validation set. You need fresh data for an honest score.
- Grouped by person: all 40 photos of a person go in the same set, so the model is tested on faces it has never seen.
- Because the parameters were chosen to fit the training rows, including their noise.
Try it on your own data: the train-test split splits a CSV at random, stratified or by time, in your browser.
New words: training set, validation set, test set, generalisation error, stratified split, grouped split, time split.
Next, Module 3: the first real model, the straight line fitted by least squares.
- Training rows fit the parameters, validation rows choose between models, and the test set gives one final honest score.
- Training error flatters the model; test error estimates how it will do on new data.
- Split at random for independent rows, stratified for rare classes, grouped for repeated people or items, and by time for anything over time.
- A small test set gives a wobbly score; its standard error shrinks like 1 ÷ √n.
Check yourself
Sources
- James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Chapter 5 (resampling methods) https://www.statlearning.com/
- Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd ed., Springer, Section 7.2 (training, validation and test) https://hastie.su.domains/ElemStatLearn/
- scikit-learn API: train_test_split https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.