Module 7: Judging models honestly · Lesson 7.4 · about 25 min
Data leakage: the most common mistake
Leakage is when a model sees during training something it will not have when used: the answer in disguise, the future, or the test rows. It makes scores too good to be true. Play spot-the-leak and learn the checklist.
Module 7 · about 25 minutes · read first: 2.5, 7.2. You’ll be able to recognise the four common kinds of leakage, check a project for them, and build a leak-proof pipeline.
1. Why this matters
Leakage is the most common reason a model that looked brilliant in testing fails in real use. It has sunk competition winners, published medical models and business launches. It is easy to cause. It is invisible in the score, which only looks better. And it is caught only by thinking about when each piece of information becomes available.
It’s like a student who saw the exam paper while revising. Great marks, no learning, and a nasty surprise at the next exam.
2. The four common kinds
- Target leakage: a column that is caused by the outcome, or recorded after it. “Number of collection calls” predicts loan default perfectly, because collection calls only happen after a default.
- Time leakage: training on the future to predict the past. For example, shuffling three years of daily sales into random folds, so the model learns from December to “predict” the March before it.
- Preprocessing leakage: fitting a scaler, an imputer (a filler for missing values), a feature selector or a target encoder on all the data before splitting. The test rows then shape what the model learns.
- Duplicate or group leakage: the same customer, patient, photo or near-identical row on both sides of the split. The test then checks memory, not learning.
1. Predicting hospital readmission. One input is "number of follow-up appointments booked in the 30 days after discharge".
Leak. Those appointments happen after the moment of prediction, and readmitted patients book more of them. At discharge, when the model is used, this column does not exist yet.
2. Before splitting into train and test, the whole dataset is scaled with its own mean and standard deviation.
Leak (a small one). The test rows helped set the mean and spread. Split first, then fit the scaler on the training rows only and apply it to the test rows.
3. Predicting next month's sales. Rows from all three years are shuffled at random into train and test.
Leak. The model trains on days after the ones it is tested on, and learns the future. With time series, train on the past and test on a later period.
4. Predicting loan default from income, debt and years at the current job, all taken from the application form.
Fine. Everything is known on the day the application arrives, which is when the model will be used.
5. Detecting pneumonia from X-rays. The same patients appear in both training and test images.
Leak. The model can recognise the patient (or their hospital's scanner) instead of the disease. Split by patient, so each person is only on one side.
6. Choosing the tree depth by cross-validation on the training set, then scoring once on a test set kept aside.
Fine. This is the right way: the test set is touched once, after every choice is made.
Watch it happen: a small preprocessing leak
Ten days of website visits (in hundreds). Two of them are the test days.
The data. Training days: 2, 4, 4, 6, 8, 6, 4, 6. Test days: 20 and 10, a busy sale weekend the model must never see.
The leak. Someone scales before splitting. The mean of all ten days is 70 ÷ 10 = 7. The two test days have dragged "normal" up from 5 to 7, and the model learns with that number.
Split first. Put the test days aside. The training mean is 40 ÷ 8 = 5. This is the only mean the model is allowed to know.
Apply, don't refit. Scale the test days with the training mean: day 20 is 15 above normal. On the day the model is used, it will see each new day this way.
3. The checklist
- For every column, ask: would I know this at the moment of prediction? If not, remove it.
- Split first: by time if the data are in time, by group if rows come in groups. Then do everything else.
- Put all preprocessing in a pipeline fitted on training rows only. scikit-learn’s
Pipelinedoes this inside cross-validation, refitting each step in every fold. - Be suspicious of any single column that is much more important than the rest (Lesson 6.4).
- Be suspicious of any score that is too good for the problem. Then go and look.
- t
- the moment the model makes its prediction in real use
- xₜ
- the inputs that are truly known at that moment
- x_j = g(y, after t)
- a column built from the answer itself, or from things that happen later: a leak
- μ (mu), σ (sigma)
- the mean and spread a scaler uses; they must come from training rows only
- Physical meaning
- A model is used in the real world on a real day, knowing only what is known that morning. Any column or step that peeks past that morning makes testing a rehearsal with the answers written on the wall.
Maths and principle
Principle: the validation set must be treated exactly like future data. Anything computed from it, or from the future, must not reach training.
In symbols: a model will be used at time with inputs known at . A training column leaks if it depends on or on events after :
Why preprocessing leaks: standardising with the mean of all rows, including the test rows, gives a training input that carries a little information about the test rows. It is usually small for scaling, and can be large for steps that use the target, such as selecting the columns most correlated with .
Analogy: a student who sees the exam paper while revising. Great marks, no learning.
Try it: predicting whether a customer will cancel next month. Which column leaks: “plan type”, “support calls in the last 3 months”, or “date of cancellation request”? (The cancellation date: it exists only for customers who already cancelled.)
“A higher score means a better model.” Not if the score was measured unfairly. When fixing a leak, the score goes down, and that is good news: the new number is the one you will actually get in use. A team that celebrates 0.97 and ships it learns the truth from angry users instead.
4. Worked example
A pattern seen in real projects, made-up numbers. A hospital model predicts pneumonia from chest X-rays with an AUC of 0.97 in testing, but 0.71 at a different hospital.
The team checks:
- Split by patient? No: 30 % of patients had images on both sides. Re-splitting by patient: AUC 0.91.
- What does it look at? Heat maps show it leans on a text marker printed in the corner of portable X-rays, used for the sickest, bedridden patients: a hint about the answer, not the disease. Cropping the corners: AUC 0.86.
The honest model is still useful, but at 0.86, not 0.97. Both leaks were invisible in the score alone.
5. Practice
- In the game, which story had a leak caused by the order of events in time?
- Why is target encoding (replacing a category by the average target for that category) risky?
- A churn model’s top feature is “account closed flag”. What do you do?
Answers
- The hospital readmission story (appointments booked after discharge) and the sales story (random shuffling across years).
- The average includes the row’s own target. Compute it on training folds only, and for each row leave the row itself out.
- Drop it: the account is closed because the customer churned. It is the answer in disguise.
Try it on your own data: the train-test split splits your CSV into training and test files before you do anything else, the first rule of the checklist.
New words: data leakage, target leakage, time leakage, preprocessing leakage, pipeline, group split.
- Leakage: the model learns from information it will not have when it is used, so its test score is too good.
- Four common kinds: target leakage, time leakage, preprocessing leakage, and duplicates or groups split across train and test.
- For every column ask “would I know this at prediction time?”; split first, then fit every preprocessing step on training rows only, inside a pipeline.
- A score that is too good for the problem is a warning, not a win.
Check yourself
Sources
- Kaufman, Rosset, Perlich and Stitelman (2012), Leakage in data mining: formulation, detection, and avoidance, ACM Transactions on Knowledge Discovery from Data 6(4), 15 https://doi.org/10.1145/2382577.2382579
- Kapoor and Narayanan (2023), Leakage and the reproducibility crisis in machine-learning-based science, Patterns 4(9), 100804 https://doi.org/10.1016/j.patter.2023.100804
- scikit-learn user guide, Common pitfalls and recommended practices (data leakage) https://scikit-learn.org/stable/common_pitfalls.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.