Module 3: Linear regression: the first model · Lesson 3.8 · about 25 min
Regularisation: ridge and lasso
Add a penalty for large weights to the error, and an over-flexible model calms down. Ridge shrinks every weight; lasso sets some to exactly zero and so picks features. Turn up the penalty and watch the weights and the test error.
Module 3 · about 25 minutes · read first: 3.4, 3.7. You’ll be able to write the ridge and lasso losses, explain why a penalty reduces overfitting, say how lasso selects features, and choose the penalty strength.
1. Why this matters
An overfitting model usually has huge weights that pull against each other to thread every training point. Regularisation adds a cost for large weights, so the model only uses big weights when the data really demands them. Think of a household budget with a tax on spending: you still buy what you truly need, but the impulse buys disappear. It is one of the most used tools in practice: in regression, in logistic regression, and in neural networks, where the same idea is called weight decay.
2. The penalised loss
Instead of minimising only the squared error, minimise error plus a penalty:
(lambda) sets the strength. At it is plain least squares. As grows, weights shrink. At a very large every weight is crushed to (almost) zero and the model predicts the mean. The bias is never penalised: shifting the whole line up or down isn’t complexity.
Scale the features first (Lesson 2.4). The penalty treats all weights alike, so a feature measured in large units (with a small weight) would barely be penalised compared with one in small units.
3. Ridge or lasso?
| Ridge (L2) | Lasso (L1) | |
|---|---|---|
| Effect on weights | Shrinks all smoothly towards zero | Shrinks, and sets the weakest to exactly zero |
| Feature selection | No | Yes: zero weight means the feature is dropped |
| Twin features (multicollinearity) | Shares the weight between them | Tends to keep one, drop the other |
| Solution | A formula: | A search (coordinate descent) |
Elastic net mixes both penalties, and is a sensible default when you aren’t sure.
Here is ridge and lasso acting on one weight, step by step. A single standardised feature has Σx² = s = 10, and plain least squares gives it weight 4:
4. Why lasso gives exact zeros
Picture just two weights. The squared error makes oval rings around the least-squares answer. The penalty limits how far from zero the weights may go: for ridge, inside a circle (); for lasso, inside a diamond (). The answer is where the smallest error ring touches the shape. A circle is round, so the touching point is usually somewhere on its curve, with both weights non-zero. A diamond has sharp corners on the axes, and the rings very often touch a corner first, where one weight is exactly zero.
Maths and principle
Principle: a penalty trades a little training error for much smaller, steadier weights, lowering variance at the cost of a little bias (Lesson 3.7).
Equations: ridge has a closed-form solution, the normal equation with added down the diagonal:
Adding to the diagonal also makes the matrix safe to invert even when features are near copies of each other, which is how ridge cures multicollinearity.
For one standardised feature, ridge simply scales the least-squares weight: , where . Lasso instead subtracts and clips: , which hits exactly zero once passes .
Analogy: a budget. Ridge taxes every pound spent on weights, so all spending shrinks a little. Lasso charges a flat fee per pound with no discount for small amounts, so tiny purchases aren’t worth it and are cut entirely.
Try it: least-squares weight 4, . Ridge with ? Lasso with , then ? (Ridge: . Lasso: , then .)
- λ (lambda)
- The price per unit of weight: 0 means no penalty, very large means every weight is crushed towards zero.
- Σ wⱼ² (ridge, L2)
- The sum of squared weights: big weights cost a lot, small ones almost nothing.
- Σ |wⱼ| (lasso, L1)
- The sum of weight sizes: every unit costs the same, so tiny weights aren't worth keeping.
- Bias w₀
- Never penalised: moving the whole prediction up or down doesn't make the model wigglier.
- Physical meaning
- Huge weights that pull against each other are how an overfitting model threads every noisy point. A penalty is a referee who says "prove you need that": a weight survives only if it cuts the error by more than it costs. What remains is a calmer model that changes less from one sample to the next.
“Regularisation makes the model fit worse, so it must make it worse.” It makes the fit on the training data slightly worse, on purpose. In return the weights stop chasing noise, so the error on new data usually drops, sometimes a lot (in the worked example, validation R² rises from 0.41 to 0.74). Pick λ by cross-validation, not by training error, which always prefers λ = 0.
5. Worked example
Made-up numbers. A retailer predicts weekly sales per store from 60 features (local population, competitors, weather, 40 product-mix shares…). With 150 stores, plain least squares overfits: training R² 0.94, validation R² 0.41.
| Model | Validation R² | Features used |
|---|---|---|
| Least squares | 0.41 | 60 |
| Ridge, best λ (by cross-validation) | 0.72 | 60, all shrunk |
| Lasso, best λ | 0.74 | 11 |
Lasso does slightly better and also tells the team which 11 features carry the signal. That makes the model easier to explain and cheaper to keep up to date, since 49 data feeds are no longer needed.
6. Practice
- What happens to the ridge weights as ?
- Two features are near copies. Which penalty is likelier to keep only one of them?
- Why isn’t the bias penalised?
Answers
- They all shrink to zero, and the model predicts the mean of y (the bias).
- Lasso.
- Moving the whole prediction up or down doesn’t make the model more wiggly; penalising it would only bias predictions towards zero.
Try it on your own data: fit a high-degree polynomial in the regression workbench and compare its R² with a simpler fit. The overfitting you see there is what a penalty tames.
New words: regularisation, penalty, λ (lambda), ridge (L2), lasso (L1), elastic net, weight decay, feature selection.
- Regularisation minimises error plus a penalty on the weights; λ sets how strong the penalty is.
- Ridge (L2) shrinks all weights smoothly; lasso (L1) shrinks and sets the weakest exactly to zero, selecting features.
- Scale the features first, never penalise the bias, and choose λ by cross-validation.
- A little extra training error buys much steadier weights and usually a lower test error.
Check yourself
Sources
- Hoerl, A. E. and Kennard, R. W. (1970), Ridge Regression: Biased Estimation for Nonorthogonal Problems, Technometrics 12(1), 55-67 https://doi.org/10.1080/00401706.1970.10488634
- Tibshirani, R. (1996), Regression Shrinkage and Selection via the Lasso, Journal of the Royal Statistical Society Series B 58(1), 267-288
- James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Section 6.2 (shrinkage methods) https://www.statlearning.com/
- scikit-learn user guide: Linear models (Ridge, Lasso, ElasticNet) https://scikit-learn.org/stable/modules/linear_model.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.