Home / Learn / Data & AI / ML foundations / Lesson 3.2
All lessons

Module 3: Linear regression: the first model · Lesson 3.2 · about 30 min

Least squares: why we square the errors

The best line is the one that makes the sum of squared errors smallest. Drag points and watch the squares, then derive the slope and intercept formulas step by step with calculus.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 3 · about 30 minutes · read first: 1.5, 3.1. You’ll be able to explain why least squares uses squared errors, derive the formulas for the best slope and intercept, and fit a line to small data by hand.

The best line is the one whose squared misses add up to the least, and calculus finds it exactly: it passes through the point of means with slope "how x and y move together ÷ how much x moves".

1. Why this matters

“The best line” needs a definition before a computer can find it. Think of a referee: before a match can be won, someone must agree what counts as a goal. Least squares, invented around 1805 by Legendre and Gauss to work out the orbits of planets and comets, is the definition almost everyone uses. Its loss, the mean squared error, comes back in neural networks, and the idea of “write the error as a function, then find its minimum” runs through the whole course.

2. Choosing what “best” means

Each point has a residual ei=yi−yˆi: positive if the point is above the line, negative if below. We want residuals small overall. Options:

  • Add them up: no good. +5 and −5 cancel, and a terrible line can score 0.
  • Add their sizes ∑|ei| (least absolute deviations): sensible and robust to outliers, but its V-shaped corners make the maths awkward.
  • Add their squares ∑ei2 (least squares): always positive, punishes a big miss much more than several small ones, and gives a smooth bowl-shaped error with a single bottom that calculus finds exactly.
Try it: drag the points and watch the squares
total squared error (area of the squares)
slope and intercept
Each square's side is one error (the gap from the point up or down to the line), so its area is that error squared. The least-squares line makes the total shaded area as small as it can be. Switch to the other lines: their squares always add up to more. Notice how one point dragged far away grows a huge square and pulls the line towards itself.

Try dragging one point far away. Its square grows with the distance squared, so least squares bends the line a long way to reduce it. That is the price of squaring: sensitivity to outliers (Lesson 2.4).

Watch least squares judge three lines through the same three points:

The data. Three days of an ice-cream van: (1, 2), (2, 4), (3, 5). The means are x̄ = 2 and ȳ = 3.67.
A flat line at the mean. Predict 3.67 every day. The misses are −1.67, 0.33 and 1.33; their squares add to 4.67. This is the "no model" baseline.
Try ŷ = 2x. Predictions 2, 4, 6. Misses 0, 0 and −1: sum of squares 1.00. Much better.
The least-squares line, ŷ = 1.5x + 0.67. Predictions 2.17, 3.67, 5.17. Misses −0.17, 0.33, −0.17: sum of squares 0.17. No other straight line does better.
A built-in check. The best line passes through the point of means (2, 3.67), and its misses add to exactly zero.

3. Deriving the best line

We want the m and c that make the sum of squared errors smallest:

S(m,c)=∑i=1n(yi−mxi−c)2

At the bottom of the bowl, the slope in every direction is zero (Lesson 1.5). So we set both partial derivatives to 0. Using the chain rule, the derivative of (yi−mxi−c)2 with respect to c is 2(yi−mxi−c)×(−1), and with respect to m it is 2(yi−mxi−c)×(−xi).

Step 1, the intercept.

∂S∂c=−2∑i(yi−mxi−c)=0⇒∑iyi−m∑ixi−nc=0⇒c=y‾−mx‾

So the best line always passes through the point of means (x‾,y‾), the “centre of gravity” of the data.

Step 2, the slope. Put c=y‾−mx‾ into the second condition:

∂S∂m=−2∑ixi((yi−y‾)−m(xi−x‾))=0

Because ∑i(yi−y‾)=0 and ∑i(xi−x‾)=0, we may replace xi in front by (xi−x‾) without changing the sum, which gives

m=∑i(xi−x‾)(yi−y‾)∑i(xi−x‾)2

The top is (n times) the covariance of x and y; the bottom is (n times) the variance of x. So the slope is “how x and y move together” divided by “how much x moves”.

Maths and principle

Principle: the least-squares line minimises the sum of squared residuals. Setting the derivatives to zero gives a formula, so no search is needed.

Equations:

m=∑(xi−x‾)(yi−y‾)∑(xi−x‾)2=rsysx,c=y‾−mx‾

The second form links the slope to the correlation r of Lesson 2.3: the slope is the correlation scaled by the ratio of spreads.

Analogy: hanging the line from springs attached to every point, where each spring pulls harder the more it is stretched. The line settles where the pulls balance.

Try it: for (1, 2), (2, 4), (3, 5), find m and c. (x‾=2, y‾=11/3; top = 3, bottom = 2, so m=1.5; c=11/3−3=2/3: yˆ=1.5x+0.67.)

eᵢ = yᵢ − ŷᵢ
The residual: how far point i is above (+) or below (−) the line.
S = Σ eᵢ²
The sum of squared errors: the score least squares makes as small as possible.
Σ(x − x̄)(y − ȳ)
How x and y move together (n times their covariance).
Σ(x − x̄)²
How much x moves on its own (n times its variance).
c = ȳ − m x̄
The intercept that makes the line pass through the point of means.
Physical meaning
Hang a stiff rod among the points and tie each point to it with a spring that pulls harder the more it is stretched. The rod settles where all the pulls balance: that resting place is the least-squares line. One point far away stretches its spring a lot and drags the rod towards it, which is why outliers matter so much.

4. Worked example

Made-up numbers. Five flats:

size x (m²) 50 60 80 100 110
price y ($000) 260 290 340 420 440
  • Means: x‾=400/5=80, y‾=1750/5=350.
  • Deviations x−x‾: −30, −20, 0, 20, 30. y−y‾: −90, −60, −10, 70, 90.
  • Products: 2700, 1200, 0, 1400, 2700. Sum = 8000.
  • Squares of x deviations: 900, 400, 0, 400, 900. Sum = 2600.
  • Slope m=8000/2600≈3.08 ($000 per m²). Intercept c=350−3.08×80≈103.8.

So price ≈ 3.08 × size + 104, in $000. A 90 m² flat: about 3.08 × 90 + 104 ≈ $381,000.

“Least squares finds the line that goes through the most points.” It may go through none of them. It balances the misses of all points at once, and squaring means it cares far more about one big miss than several small ones. If you want a line that one wild point can’t drag around, you need a different loss, such as least absolute deviations.

5. Practice

  1. Fit a line by least squares to (0, 1), (1, 3), (2, 5), (3, 7). What do you notice?
  2. Show that the residuals of a least-squares line (with an intercept) always add up to 0.
  3. If every y is doubled, what happens to m and c?
Answers
  1. The points lie exactly on y=2x+1, so least squares returns it with every residual 0.
  2. That is the condition ∂S/∂c=0: −2∑(yi−mxi−c)=0, so ∑ei=0.
  3. Both double: the top of the slope formula doubles, and c=y‾−mx‾ doubles too.

Try it on your own data: the best-fit line tool draws the least-squares line through your own points and shows the working.

New words: least squares, sum of squared errors, point of means, least absolute deviations.

  • The residual is the gap between a point and the line; least squares makes the sum of squared residuals smallest.
  • Squaring keeps every miss positive and gives a smooth bowl with one bottom, but makes the line sensitive to outliers.
  • Best slope m = Σ(x − x̄)(y − ȳ) ÷ Σ(x − x̄)²; intercept c = ȳ − m x̄.
  • The least-squares line passes through the point of means and its residuals add to zero.

Check yourself

1Why are the errors squared rather than just added?
2For x = 1, 2, 3 and y = 2, 4, 5, the least-squares slope is:
3The least-squares line always passes through which point?

Sources

  • James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Chapter 3 (linear regression) https://www.statlearning.com/
  • Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd ed., Springer, Chapter 3 (linear methods for regression) https://hastie.su.domains/ElemStatLearn/
  • Stigler, S. M. (1986), The History of Statistics: The Measurement of Uncertainty before 1900, Harvard University Press, Chapter 1 (Legendre, Gauss and least squares)

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.