Module 4: Classification and logistic regression · Lesson 4.4 · about 25 min
Decision boundaries
A classifier splits the space of inputs into regions, one per class. Logistic regression draws a straight boundary; add squared features and it can draw circles. Watch a boundary learn live, add your own points, and meet the XOR problem.
Module 4 · about 25 minutes · read first: 4.2, 4.3. You’ll be able to find a logistic regression’s decision boundary, explain when a straight boundary is not enough, and fix it with features.
1. Why this matters
A picture of the decision boundary (the border between the regions where a model says “yes” and “no”) is the quickest way to see what a classifier can and can’t do. Every classifier in this course, from k-nearest neighbours to neural networks, can be compared by the border shapes it can draw. It is also how the “AI winter” of the 1970s began: a single neuron couldn’t learn a simple pattern called XOR (Lesson 9.1).
2. Where the boundary is
Logistic regression predicts class 1 when . Because , that is the same as the score . So the boundary is the set of points where the score is exactly zero:
With two inputs that is a straight line. With three it is a flat sheet (a plane), and with more a hyperplane, the same idea in more dimensions. The weight vector points straight across the line, towards class 1. The farther a point is from the line, the surer the model is.
The model. A loan model uses income x₁ and debt x₂ (both standardised: 0 is average). Its score is z = 1.8x₁ − 2.4x₂ + 0.6.
Find the border. Set z = 0 and solve for debt: x₂ = 0.75x₁ + 0.25. That straight line is the decision boundary.
Pick a side. Below the line (less debt for the income) z is positive: "repays". Above it, "defaults".
Test an applicant. Average income and average debt, (0, 0): z = 0.6, p = σ(0.6) ≈ 0.65. Just on the "repays" side.
Farther means surer. An applicant at (1, −1) gets z = 1.8 + 2.4 + 0.6 = 4.8, p ≈ 0.99. Deep inside the "repays" region.
- w·x + b = 0
- The decision boundary: every point where the model is exactly 50/50.
- w
- The weights. Together they form an arrow pointing across the boundary towards class 1.
- ‖w‖
- The length of that arrow: how quickly the probability changes as you cross the line.
- Physical meaning
- A fence across a field. Plain logistic regression can only build straight fences. Feature tricks let it build curved fences by first redrawing the map.
3. Watch it learn
Try:
- Two groups: a straight boundary settles between them within a few hundred steps.
- Add a few circles deep in the square region (tap with “a circle” selected): the boundary tilts to compromise. One straight line can’t please everyone.
- One group inside a ring: no straight line works, and the model stays unsure (pale shading, accuracy near 50 %). Switch on the squared features and it draws an oval round the inner group.
- Four corners (XOR): also impossible for a line. The product feature is exactly what it needs, because is positive in two opposite corners and negative in the other two.
4. Straight in what?
The model with squared features is
It is still logistic regression, trained the same way, and still a straight-line sum of its weights. In the five-dimensional space of the new features, the boundary is a flat hyperplane. Seen back in the original two dimensions, it is a curve. Picture drawing on a sheet of paper, then bending the paper: a straight line on the bent sheet can look curved from above. This idea, “remap the data so a straight boundary works”, is the heart of kernel methods and, in a learned form, of neural networks, whose hidden layers find the new features by themselves (Lesson 9.3).
“Logistic regression can only ever draw straight lines, so it is useless for curved patterns.” It draws straight lines in whatever features you give it. Give it squares and products of the inputs and its boundary in the original inputs becomes a circle, an oval or a saddle. What is fixed is that the score is a weighted sum; what the sum is made of is up to you.
Maths and principle
Principle: a linear classifier’s boundary is the hyperplane . Data that some hyperplane splits perfectly is called linearly separable.
Distance from the boundary: for a point , the signed distance to the boundary is
so the score is distance times . A longer makes the probability climb faster with distance: a steeper S across the line.
Try it: , . Which class for the point (1, 1)? For (−1, 1)? (: class 1. : class 0.)
5. Worked example
Made-up numbers. A loan model uses income and existing debt , both standardised, to predict repayment: .
- Boundary: . Above that line (more debt for the income), the model predicts default.
- An applicant at average income and average debt (): , of repaying. On the “repays” side, but not by much.
- The ratio of the weights, 1.8 / 2.4 = 0.75, says one standard deviation more debt is balanced by 1.33 standard deviations more income.
6. Practice
- Find the boundary of and say which side is class 1.
- Which data in the simulation is linearly separable?
- Why does the model stay pale (unsure) on the ring data with plain features?
Answers
- The line ; class 1 is above and to the right, where .
- The two groups (apart from the odd overlapping point).
- No straight line helps, so the best it can do is keep near zero and predict about the base rate everywhere.
- The decision boundary is where the model is exactly 50/50; for logistic regression that is w·x + b = 0.
- With plain inputs the boundary is a straight line (a hyperplane in more dimensions).
- Data a straight line can split perfectly is called linearly separable; XOR and rings are not.
- Adding squares and products of inputs lets the same model draw curved boundaries.
Try it on your own data: the small model trainer trains logistic regression on your own CSV; add a product of two columns as an extra column to give it a curved boundary.
New words: decision boundary, hyperplane, linearly separable, XOR problem, feature map.
Check yourself
Sources
- James, Witten, Hastie and Tibshirani (2021), An Introduction to Statistical Learning, 2nd edition, Springer, Chapters 4 and 9 https://www.statlearning.com/
- Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning, 2nd edition, Springer, Section 4.4 (Logistic regression) and 4.5 (Separating hyperplanes)
- Minsky and Papert (1969), Perceptrons: An Introduction to Computational Geometry, MIT Press (the XOR limit of a single layer)
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.