Module 3: Linear regression: the first model · Lesson 3.6 · about 20 min
Curves with a linear model: polynomial features
Add x², x³ and other new columns and a linear model can draw curves, because it is linear in its weights, not in x. Raise the degree of a polynomial and watch it bend, then learn the other useful features: logs, interactions and bins.
Module 3 · about 20 minutes · read first: 3.1, 3.4. You’ll be able to fit curves with a linear model by adding polynomial features, and use log, interaction and binned features to capture other shapes.
1. Why this matters
Lesson 3.5 showed a U-shaped residual plot: the data curves and a straight line can’t. You don’t need a new kind of model to fix that. You need new features: extra columns made from the ones you have. This trick, called feature engineering, is often the cheapest way to improve any model. It is like giving a carpenter who only owns a straight saw some pre-bent pieces of wood: the tool stays the same, but the furniture can now curve.
2. Powers of x
Make columns and fit an ordinary multiple regression on them:
The prediction is still a weighted sum, so everything from Lessons 3.2 to 3.4 applies: the normal equation, gradient descent, the scores. Only the picture changes: a parabola for degree 2, an S for degree 3, and more wiggles as the degree rises.
Try degree 1, then 3, then 12. Degree 3 follows the curve well. Degree 12 threads through every training point but swings wildly between them: it has started to model the noise. The next lesson measures this.
“Adding x² makes it a non-linear model, so all the linear tools stop working.” The curve is non-linear in x, but the model is still a weighted sum of its columns, so it is linear in its weights. The normal equation, gradient descent and every score from Lessons 3.2 to 3.5 work unchanged. You only changed the columns.
Practical note: powers of big numbers explode (100¹⁰ is 10²⁰), which ruins the arithmetic. Scale x to roughly −1 to 1 before taking powers, as the simulation does.
3. Other useful features
| Feature | Made from | Catches |
|---|---|---|
| Log or predicting | Money, populations, sizes | Effects that are percentages, long right tails |
| Interaction | Two inputs | One input’s effect depending on another (ads work better at weekends) |
| Bins (one-hot of ranges) | Age into 18-25, 26-40, … | Steps and odd shapes without picking a formula |
| Ratio | Price and size | Price per m², debt-to-income |
| Time parts | Dates | Hour, weekday, season (Lesson 2.1) |
| Sine and cosine | Cycles | Hour 23 being next to hour 0 |
Domain knowledge is the best source of features. A doctor knows body mass index (weight ÷ height²) matters more than either alone; an analyst knows “days since last order” predicts cancelling better than the order date itself.
Maths and principle
Principle: a model is linear in its weights if the prediction is a weighted sum. The inputs to that sum may be any fixed functions of the raw data, , called basis functions or features.
Equation:
The design matrix (Lesson 3.4) simply has columns , and the normal equation is unchanged.
Counting: with raw inputs and all products up to degree 2, the number of features is . For 10 inputs that is 66; for 100, 5,151. Features multiply fast, which is one reason to prefer models that find interactions by themselves (trees, Module 5).
Analogy: a straight ruler can’t trace a curve, but a set of flexible rulers glued together at chosen bends can. Each new feature is one more bend you allow.
Try it: fit exactly through (−1, 2), (0, 1), (1, 2). (From x = 0: . Then and , so , : .)
- φⱼ(x)
- A feature made from the raw input, such as x², log x or x₁ × x₂.
- wⱼ
- Its weight, learned exactly as before.
- Degree
- The highest power used. Each extra degree lets the curve bend once more.
- Physical meaning
- Real effects bend. Iced drinks sell better as it gets hotter, until it is so hot that people stay indoors; a second year of experience adds more to pay than a twentieth. A squared column lets a model say "rising, then levelling off, then falling" in one tidy formula that a person can still read.
4. Worked example
Made-up numbers. A café’s daily iced-drink sales rise with temperature up to about 30 °C, then fall a little on the hottest days (people stay indoors). A straight line on temperature gives R² = 0.71 with an arch in the residuals. Adding :
- At 20 °C: −180 + 420 − 140 = 100 drinks.
- At 30 °C: −180 + 630 − 315 = 135 drinks.
- At 36 °C: −180 + 756 − 453.6 ≈ 122 drinks.
The peak is where the slope is zero: °C. Test R² rises to 0.86 and the residual arch disappears. The fitted curve says something useful in plain words, which a wiggly degree-12 curve never would.
5. Practice
- How many weights has a degree-4 polynomial in one input?
- Sales grow by about 8 % for each extra store opened. What feature or target change makes this linear?
- Why scale x before computing ?
Answers
- Five (powers 0 to 4).
- Predict log(sales): an 8 % rise each time is a constant step of log(1.08) ≈ 0.077.
- Large x makes huge numbers that swamp the arithmetic and make the normal equation unstable; small x keeps all powers in a sensible range.
Try it on your own data: the regression workbench fits polynomial regression, and the curve fitter fits curves of any formula you choose.
New words: feature engineering, polynomial features, basis function, interaction term, log transform, binning.
- Feature engineering makes new columns from old ones; the model stays a weighted sum, so all linear tools still work.
- Powers of x let a linear model fit curves; degree 2 gives a parabola with one bend.
- Logs, interactions, bins, ratios and sine/cosine catch percentages, combined effects, steps and cycles.
- Scale x before taking high powers, and keep the degree low: high degrees chase noise.
Check yourself
Sources
- James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Chapter 7 (moving beyond linearity) https://www.statlearning.com/
- Hastie, T., Tibshirani, R. and Friedman, J. (2009), The Elements of Statistical Learning, 2nd ed., Springer, Chapter 5 (basis expansions) https://hastie.su.domains/ElemStatLearn/
- scikit-learn API: PolynomialFeatures https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.