All lessons

Module 8: Relationships and business data · Lesson 8.2 · about 25 min

Regression for explaining: coefficients, intervals and p-values

Read a regression table like an analyst: what a coefficient means, its standard error and confidence interval, the t-test for "no effect", R², controlling for other variables, and the assumptions to check in the residuals.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 8 · about 25 minutes · uses Lessons 5.1 and 6.3. You’ll be able to read every number in a regression table, put an interval on a coefficient, and say what it does and doesn’t show.

A regression coefficient is the expected change in the outcome for a one-unit change in a predictor, holding the others fixed, with its own standard error and interval.

In everyday terms: a recipe that says “each extra spoon of sugar makes the cake about 5 % sweeter”. The coefficient is that “5 % per spoon”; its interval says how sure the baker is.

1. Predicting vs explaining

ML foundations taught regression for prediction (least squares, measuring fit): does it forecast well on new data? Analysts also use it for explaining: how much does Y change with X, and how sure are we? That needs standard errors, intervals and tests.

Try it: how sure is the slope?
fitted slope (95 % CI)
t, p for slope = 0
R²
Each dot is a week: ad spend ($) and extra sales ($). The dark line is the least-squares fit (ML foundations Lesson 3.2); the pale lines are slopes picked from inside its 95 % interval. More weeks or less noise narrow the fan. A slope can be clearly real while R² stays low: noise hides individual weeks, not the trend. Made-up numbers.

With 20 weeks and noise 25, the slope’s interval is wide. Push the weeks to 100: same true line, much narrower fan. Then raise the noise: R² falls, but with enough weeks the slope stays clearly above zero.

2. Reading the table

Term Coefficient Std. error t p 95 % CI
Intercept 120 28 4.3 0.0004 61 to 179
Ad spend ($) 1.60 0.55 2.91 0.009 0.44 to 2.76
  • Coefficient: the average change in Y for one more unit of X.
  • Standard error: how much the coefficient would vary from sample to sample.
  • t = coefficient ÷ SE, compared with a t-distribution with n−k−1 degrees of freedom (k = number of inputs).
  • 95 % CI: coefficient ± t-critical × SE.
  • Intercept: the predicted Y when every X is 0; often meaningless (zero ad spend may be outside the data).

3. Controlling for other variables

With several inputs, each coefficient is the change in Y for one more unit of that X holding the others fixed. Adding a confounder (store size) can shrink or even flip a coefficient (next lesson). It’s a powerful tool, but only for confounders you’ve measured.

Categorical inputs enter as 0/1 dummy columns; each coefficient is the difference from a baseline category.

4. Check the assumptions

Look at the residuals (actual − fitted):

  • Straight line? A curve in residuals vs fitted means the relationship isn’t linear (try log or a squared term).
  • Even spread? A funnel shape (bigger errors for bigger values) makes SEs wrong; use robust SEs or log the outcome.
  • Independent? Weekly data often has runs of positive and negative residuals; standard SEs are then too small.
  • Outliers and leverage: one extreme point can move the line a lot (Lesson 8.1).
Coefficient b
Change in y per one-unit change in x, other predictors held fixed.
SE(b)
How much b would wobble across samples.
t = b ÷ SE(b)
Is b distinguishable from 0?
R²
Share of variation in y explained by the model.
Physical meaning
The slope is a rate of exchange between two quantities in the data: dollars of sales per dollar of ads. It describes how they moved together in the past, not what would happen if you changed one of them.

“The coefficient shows what happens if we spend more.” Observational regression shows association. If ad spend rose in weeks the team expected to be busy, the slope mixes the ads’ effect with the busy weeks. Experiments answer “what if”.

Watch it happen, one step at a time:

The data. 20 weeks of ad spend and sales, both in $100s.
Slope. b = 1.60: each extra $100 of ads went with about $160 more sales.
Uncertainty. SE ≈ 0.55, t ≈ 2.9, 95 % CI $44 to $276 per $100.
Fit. R² = 0.32: ads line up with a third of the week-to-week variation.
Limits. One holiday week sits high; without it the slope is 1.45. Causation needs a geo experiment.
Maths and principle

Principle: the slope is a weighted average of the data, so it has a sampling distribution like a mean, with SE shrinking like 1/n.

Simple regression:

βˆ1=∑(xi−x‾)(yi−y‾)∑(xi−x‾)2,SE(βˆ1)=s∑(xi−x‾)2

where s=SSE/(n−2) is the typical residual. Spreading the x values wider shrinks the SE, which is why tests with a wide range of ad spend learn more.

Check with the table: t = 1.60 / 0.55 = 2.91 with 18 df, p = 0.009; t-critical 2.101, so CI 1.60 ± 1.16 = 0.44 to 2.76 (scipy agrees; tests/stats.mjs checks regress against scipy’s linregress).

5. Worked example

Made-up data: 20 weeks of ad spend and sales, both measured in hundreds of dollars, as in the table.

  • “Each extra $100 of weekly ad spend went with about $160 more weekly sales (95 % CI $44 to $276).”
  • R² = 0.32: ads track about a third of the week-to-week variation.
  • Residuals vs fitted: no curve, no funnel; one holiday week sits high. Re-fitting without it: slope 1.45. Mention it.
  • What it doesn’t show: that ads caused the sales. Ad spend may have risen in weeks the team expected to be busy. A geo experiment (ads on in some regions, off in others) would answer that.

6. Practice

  1. Coefficient 0.8, SE 0.5, n = 30, one input. Is the slope significant at 0.05?
  2. Why might the intercept be meaningless?
  3. A funnel shape appears in the residuals. Name two remedies.
Answers
  1. t = 1.6 with 28 df, p ≈ 0.12. No; the CI (about −0.22 to 1.82) includes 0.
  2. X = 0 may lie far outside the data (no store has zero floor space), so the line’s value there is an extrapolation.
  3. Model log(Y) instead of Y; use robust (heteroscedasticity-consistent) standard errors.

Try it on your own data: the regression workbench tool gives the full table with standard errors, intervals, residual plots and ANOVA; curve fitter handles curved relationships.

New words: coefficient, standard error of a coefficient, dummy variable, controlling for, residual plot, heteroscedasticity, robust standard errors.

  • A coefficient is the change in y per unit of x, holding other predictors fixed.
  • Report it with its CI; t = b ÷ SE tests “no association”.
  • R² measures fit, not importance or causation.
  • Check residual plots for curves, funnels and outliers.

Check yourself

1Sales = 120 + 1.6 × ad spend. What does 1.6 mean?
2The slope is 1.6 with 95 % CI 0.44 to 2.76. What can you say?
3R² is 0.15 but the slope's p-value is 0.001. Is that a contradiction?

Sources

  • OpenIntro Statistics, 4th edition (Diez, Cetinkaya-Rundel, Barr), Chapters 8-9 Linear and multiple regression https://www.openintro.org/book/os/
  • James, G., Witten, D., Hastie, T., Tibshirani, R. and Taylor, J. (2023), An Introduction to Statistical Learning with Applications in Python, Springer, Chapter 3 https://www.statlearning.com/

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.