Module 3: Linear regression: the first model · Lesson 3.2 · about 30 min
Least squares: why we square the errors
The best line is the one that makes the sum of squared errors smallest. Drag points and watch the squares, then derive the slope and intercept formulas step by step with calculus.
Module 3 · about 30 minutes · read first: 1.5, 3.1. You’ll be able to explain why least squares uses squared errors, derive the formulas for the best slope and intercept, and fit a line to small data by hand.
1. Why this matters
“The best line” needs a definition before a computer can find it. Think of a referee: before a match can be won, someone must agree what counts as a goal. Least squares, invented around 1805 by Legendre and Gauss to work out the orbits of planets and comets, is the definition almost everyone uses. Its loss, the mean squared error, comes back in neural networks, and the idea of “write the error as a function, then find its minimum” runs through the whole course.
2. Choosing what “best” means
Each point has a residual : positive if the point is above the line, negative if below. We want residuals small overall. Options:
- Add them up: no good. +5 and −5 cancel, and a terrible line can score 0.
- Add their sizes (least absolute deviations): sensible and robust to outliers, but its V-shaped corners make the maths awkward.
- Add their squares (least squares): always positive, punishes a big miss much more than several small ones, and gives a smooth bowl-shaped error with a single bottom that calculus finds exactly.
Try dragging one point far away. Its square grows with the distance squared, so least squares bends the line a long way to reduce it. That is the price of squaring: sensitivity to outliers (Lesson 2.4).
Watch least squares judge three lines through the same three points:
3. Deriving the best line
We want the and that make the sum of squared errors smallest:
At the bottom of the bowl, the slope in every direction is zero (Lesson 1.5). So we set both partial derivatives to 0. Using the chain rule, the derivative of with respect to is , and with respect to it is .
Step 1, the intercept.
So the best line always passes through the point of means , the “centre of gravity” of the data.
Step 2, the slope. Put into the second condition:
Because and , we may replace in front by without changing the sum, which gives
The top is (n times) the covariance of x and y; the bottom is (n times) the variance of x. So the slope is “how x and y move together” divided by “how much x moves”.
Maths and principle
Principle: the least-squares line minimises the sum of squared residuals. Setting the derivatives to zero gives a formula, so no search is needed.
Equations:
The second form links the slope to the correlation of Lesson 2.3: the slope is the correlation scaled by the ratio of spreads.
Analogy: hanging the line from springs attached to every point, where each spring pulls harder the more it is stretched. The line settles where the pulls balance.
Try it: for (1, 2), (2, 4), (3, 5), find and . (, ; top = 3, bottom = 2, so ; : .)
- eᵢ = yᵢ − ŷᵢ
- The residual: how far point i is above (+) or below (−) the line.
- S = Σ eᵢ²
- The sum of squared errors: the score least squares makes as small as possible.
- Σ(x − x̄)(y − ȳ)
- How x and y move together (n times their covariance).
- Σ(x − x̄)²
- How much x moves on its own (n times its variance).
- c = ȳ − m x̄
- The intercept that makes the line pass through the point of means.
- Physical meaning
- Hang a stiff rod among the points and tie each point to it with a spring that pulls harder the more it is stretched. The rod settles where all the pulls balance: that resting place is the least-squares line. One point far away stretches its spring a lot and drags the rod towards it, which is why outliers matter so much.
4. Worked example
Made-up numbers. Five flats:
| size x (m²) | 50 | 60 | 80 | 100 | 110 |
|---|---|---|---|---|---|
| price y ($000) | 260 | 290 | 340 | 420 | 440 |
- Means: , .
- Deviations : −30, −20, 0, 20, 30. : −90, −60, −10, 70, 90.
- Products: 2700, 1200, 0, 1400, 2700. Sum = 8000.
- Squares of x deviations: 900, 400, 0, 400, 900. Sum = 2600.
- Slope ($000 per m²). Intercept .
So price ≈ 3.08 × size + 104, in $000. A 90 m² flat: about 3.08 × 90 + 104 ≈ $381,000.
“Least squares finds the line that goes through the most points.” It may go through none of them. It balances the misses of all points at once, and squaring means it cares far more about one big miss than several small ones. If you want a line that one wild point can’t drag around, you need a different loss, such as least absolute deviations.
5. Practice
- Fit a line by least squares to (0, 1), (1, 3), (2, 5), (3, 7). What do you notice?
- Show that the residuals of a least-squares line (with an intercept) always add up to 0.
- If every y is doubled, what happens to m and c?
Answers
- The points lie exactly on , so least squares returns it with every residual 0.
- That is the condition : , so .
- Both double: the top of the slope formula doubles, and doubles too.
Try it on your own data: the best-fit line tool draws the least-squares line through your own points and shows the working.
New words: least squares, sum of squared errors, point of means, least absolute deviations.
- The residual is the gap between a point and the line; least squares makes the sum of squared residuals smallest.
- Squaring keeps every miss positive and gives a smooth bowl with one bottom, but makes the line sensitive to outliers.
- Best slope m = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)²; intercept c = ȳ − m x̄.
- The least-squares line passes through the point of means and its residuals add to zero.
Check yourself
Sources
- James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Chapter 3 (linear regression) https://www.statlearning.com/
- Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd ed., Springer, Chapter 3 (linear methods for regression) https://hastie.su.domains/ElemStatLearn/
- Stigler, S. M. (1986), The History of Statistics: The Measurement of Uncertainty before 1900, Harvard University Press, Chapter 1 (Legendre, Gauss and least squares)
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.