Home / Learn / Data & AI / ML foundations / Lesson 11.2
All lessons

Module 11: Capstone and next steps · Lesson 11.2 · about 20 min

Which model when

A one-page guide to choosing a model: what each one is good at, what it needs, and how to explain it. Answer four questions for a suggested starting point, and learn the rule for picking the simpler of two close models.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 11 · about 20 minutes · read first: 7.1, 7.2. You’ll be able to choose a sensible first model for a new problem, name a stronger one to compare against, and use the one-standard-error rule to pick between close models.

Start simple, compare fairly, and when two models are about as good, take the simpler one.

1. Why this matters

The most common question after a course like this is “which model should I use?”. The honest answer: start simple, compare fairly, and pick the simplest model that is good enough. This lesson turns the whole course into a short guide you can come back to.

2. The short guide

job start with then compare explain with
predict a number linear regression random forest, boosted trees weights; importance
yes/no or category logistic regression random forest, boosted trees weights (odds); importance
must explain each case logistic or shallow tree a pruned tree weights; the tree’s rules
find groups k-means (scaled) hierarchical clustering cluster averages
too many columns PCA — loadings
unusual rows z-scores neighbour distance which column is extreme
text, images, sound a pretrained model fine-tune a network examples; attention
Try it: which model should I start with?
Whatever the suggestion: split off test data first (Lesson 2.5), beat a simple baseline (the average, or the most common class), compare models with cross-validation (Lesson 7.2), and watch for leakage (Lesson 7.4). On tables of columns, gradient-boosted trees win most competitions; on text, pictures and sound, neural networks do. Simple models are often close enough, and far easier to explain.

Some rules of thumb:

Everyday picture: a toolbox. You reach for the screwdriver before the power drill, and only bring out the drill when the screwdriver really isn’t enough.

  • Small data (hundreds of rows): simple models, strong regularisation, cross-validation over a single split.
  • Tables of columns: trees and boosting usually beat neural networks.
  • Text, images and sound: start from a pretrained network, never from scratch.
  • When every prediction must be explained to a customer or regulator, the explanation is part of the job: choose a model whose reasoning can be read.
  • Always beat a baseline, keep a test set, and look for leakage.
Maths and principle

Principle: among models whose scores are within chance of each other, take the simplest.

One-standard-error rule. With k-fold cross-validation, each model has a mean error e‾ and a standard deviation s across the folds. The standard error of the mean is

SE=sk

Find the model with the lowest e‾min⁡. Then choose the simplest model with

e‾≤e‾min⁡+SEmin⁡

Analogy: buying a car. If two cars score the same within the test’s margin, you take the cheaper, simpler one to run.

Try it: 10 folds, the best model’s error is 0.120 with fold standard deviation 0.020. A simpler model scores 0.125. Within one SE? (SE = 0.020/10=0.0063; limit 0.1263; yes, choose the simpler one.)

k
the number of cross-validation folds: how many times each model is tested on a different slice
ē
a model's mean error over the k folds
s
how much that error jumps about from fold to fold (standard deviation)
SE = s / √k
the standard error: how much the mean error itself could shift just by chance
ēmin + SEmin
the limit: any model at or under it is as good as the best, within chance
Physical meaning
The rule stops you paying for a difference that may be luck. A simpler model is cheaper to run, easier to explain to a customer or regulator, and usually steadier on new data, so it wins any tie that the test can't separate.

3. Worked example

Watch the rule pick a model:

Mean errors. Five-fold cross-validation gives logistic regression 0.180, a random forest 0.176 and gradient boosting 0.170. Boosting looks best.

How sure are we? Each model's fold spread divided by √5 gives its standard error: 0.009, 0.011 and 0.013. The bars overlap a lot.

Draw the limit. Best mean + its standard error: 0.170 + 0.013 = 0.183.

Who is inside? 0.180 and 0.176 are both under 0.183, so all three are as good as each other within chance.

Take the simplest. Logistic regression: fast, cheap, and each weight can be shown to the customer team.

A bank team predicts which customers will leave, with 5-fold cross-validation on 12,000 rows:

model mean error fold sd SE = sd/√5
logistic regression 0.180 0.020 0.009
random forest 0.176 0.025 0.011
gradient boosting 0.170 0.030 0.013

The best is boosting, 0.170. Its limit is 0.170 + 0.013 = 0.183. Logistic regression (0.180) is inside it, so the one-standard-error rule picks logistic regression: as good within chance, fast, and each weight can be shown to the customer team. If the business later values every last bit of accuracy, boosting is the documented next step.

“The most advanced model is always the best choice.” On tables of columns, boosted trees usually beat deep networks, and on small data a straight line often beats both. A transformer trained from scratch on 5,000 rows is slower, harder to explain and usually worse. Choose for the data and the job, not the headline.

4. Practice

  1. In the chooser, pick “Predict a number”, more than 100,000 rows, and no need to explain. What does it suggest, and why?
  2. Why is “a transformer trained from scratch” almost never the right first step at work?
  3. In the worked example, would the random forest also be within the limit? Why would you still prefer logistic regression?
Answers
  1. Linear regression as the baseline, then gradient-boosted trees: with plenty of rows and no explanation needed, the most accurate table model is worth trying.
  2. It needs vast data and computing. A pretrained model, adapted, is cheaper and almost always better.
  3. Yes (0.176 is under 0.183), but logistic regression is simpler and easier to explain, and the rule picks the simplest model inside the limit.

Try it on your own data: the small model trainer fits a decision tree, logistic regression or k-nearest neighbours to your own CSV, with train and test scores side by side, so you can compare them on the same split.

New words: baseline, one-standard-error rule, standard error, pretrained model, model selection, interpretability.

  • Start with a simple baseline (linear or logistic regression), then compare a stronger model such as boosted trees.
  • Tables of columns: trees and boosting; text, images and sound: a pretrained model, adapted.
  • When each decision must be explained, choose a model whose reasoning can be read.
  • One-standard-error rule: SE = s ÷ √k; take the simplest model within best mean + SE.
  • Always keep a test set, beat a baseline and look for leakage, whatever the model.

Check yourself

1For a table of columns with 50,000 rows, where accuracy matters most, a strong default is:
2Two models' cross-validated errors differ by less than one standard error. Which do you choose?
3You must explain every loan decision to the customer. A good first choice is

Sources

  • Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd edition, Springer, section 7.10 (cross-validation and the one-standard-error rule) https://hastie.su.domains/ElemStatLearn/
  • James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, chapters 5 and 6 https://www.statlearning.com/
  • Grinsztajn, L., Oyallon, E. and Varoquaux, G. (2022), Why do tree-based models still outperform deep learning on typical tabular data? https://arxiv.org/abs/2207.08815

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.