Home / Learn / Data & AI / ML foundations / Lesson 6.3
All lessons

Module 6: Ensembles: random forests and boosting · Lesson 6.3 · about 25 min

Random forests

A random forest grows many deep trees, each on a bootstrap sample and each split choosing from a random handful of columns, then lets them vote. Add trees one by one and watch the jagged boxes smooth into a good boundary.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 6 · about 25 minutes · read first: 6.2. You’ll be able to explain how a random forest is built and why the random columns matter, choose its main settings, and read its out-of-bag score.

A random forest is a crowd of deep trees, each shown different rows and, at every question, only a few of the columns, so they disagree in useful ways and their vote is steady.

1. Why this matters

The random forest (Leo Breiman, 2001) is the most dependable first model for table data. It needs almost no tuning, handles numbers and categories, copes with useless columns and odd scales, rarely overfits badly, and gives a built-in test score. Many data teams try a random forest before anything else, and keep it more often than you’d expect.

2. Two kinds of randomness

A random forest is bagged trees (Lesson 6.2) plus one more trick:

  1. Random rows: each tree is grown on its own bootstrap sample.
  2. Random columns: at every split, the tree may only choose among a random handful of m columns (a fresh handful at each split).

Then all trees vote (classification) or are averaged (regression). The trees are grown deep, with no pruning: each one overfits in its own way, and the vote cancels it out.

Why the random columns? With bagging alone, if one column is very strong, every tree puts it at the top and the trees end up alike. Their mistakes are then correlated, and averaging helps less (Lesson 6.1). Hiding the strong column from some splits forces trees to find other routes, and the forest gets more varied. It is like a quiz team where each member is only allowed to study some of the topics: nobody can copy the best student, so they learn different things.

Try it: add trees to the forest
one deep tree: test accuracy
this forest: test accuracy
With 1 tree you see one deep, overfitted tree: hard-edged boxes round noisy points. Each new tree saw a slightly different sample and different columns, so it makes different mistakes; averaging their votes cancels much of the noise. The shading becomes a smooth probability, and test accuracy climbs, then levels off. More trees never cause overfitting; they only cost time.

Tree 1 gets its own bootstrap sample of the customers. At its first split it may only look at 4 of the 16 columns, say tenure, region, age and plan.

It picks the best of those four (tenure), splits, and at the next split draws a new random 4. It keeps going until its leaves are pure.

Trees 2 to 300 do the same with their own samples and their own random columns. The strongest column, contract type, is missing from about three quarters of the first splits, so many trees start somewhere else.

A new customer arrives. 210 trees say "leaves", 90 say "stays". The forest predicts "leaves" with probability 210 ÷ 300 = 0.70.

B
The number of trees in the forest.
m
How many randomly chosen columns each split may look at (often √p for p columns).
P̂(y = c | x)
The forest's probability for class c: the average of the trees' answers, which is close to the share of trees voting for c.
Physical meaning
A panel of experts who each studied a different selection of case files and were each told to ignore some of the evidence. None is the best expert, but their vote beats any one of them, because their blind spots are in different places.

3. The settings that matter

  • Number of trees. More is never worse, only slower. A few hundred is usual; stop when the out-of-bag score stops improving.
  • Columns per split, m. The main tuning dial. A common default is p of the p columns for classification. Smaller m means more varied but individually weaker trees.
  • Leaf size or depth. Usually left deep; a slightly larger smallest leaf can help on noisy data.

Things a forest does not do well: extrapolate beyond the training range (it is made of trees, Lesson 5.4), explain itself in one picture (a hundred trees is not a rule you can print), or handle very wide, sparse data such as raw text as well as linear models do.

“Too many trees will make a random forest overfit.” Adding trees doesn’t make any single tree more complex; it only makes the average steadier. Accuracy rises and then levels off. What can overfit is the individual tree settings (leaves that are too small on very noisy data), not the number of trees. Extra trees cost only time.

Maths and principle

Principle: average many low-bias, high-variance trees, and make them as uncorrelated as possible so the average removes most of the variance.

The forest’s prediction with trees T1,…,TB: for regression the mean, for classification the average share for each class,

Pˆ(y=c|𝐱)=1B∑b=1BPˆb(y=c|𝐱)

Why fewer columns help: from Lesson 6.1, the averaged prediction has variance ρσ2+(1−ρ)σ2/B. As B grows the second part vanishes; what remains, ρσ2, falls as the trees become less correlated, and random columns lower ρ.

Try it: with 25 columns in a classification problem, how many does each split look at by default? (25=5.)

4. Worked example

Made-up numbers. A telecom company predicts which customers will leave, with 16 columns and 20,000 rows.

model test accuracy out-of-bag accuracy
one deep tree 84.1 % (none)
bagging, 300 trees, all 16 columns per split 88.6 % 88.4 %
random forest, 300 trees, 4 columns per split 89.9 % 89.7 %

The random columns add 1.3 points over plain bagging. The out-of-bag score lands within 0.2 points of the test score, so for tuning the forest the team could have skipped the separate validation set.

5. Practice

  1. In the simulation, compare 1 tree with 60. What happens to the shading and the test accuracy?
  2. Why are the forest’s trees grown deep instead of pruned?
  3. Your forest has 500 trees and you raise it to 2,000. What do you expect?
Answers
  1. The hard boxes become a smooth gradient of vote shares, and test accuracy rises by several points before levelling off.
  2. Deep trees have low bias; their high variance is what the averaging removes. Pruned trees would add bias that averaging can’t fix.
  3. Almost the same accuracy, four times the training and prediction time.
  • A random forest = bootstrap samples (random rows) + a random handful of columns at every split + a vote.
  • The random columns make the trees less alike, so averaging removes more of their errors.
  • More trees never hurt accuracy, only speed; the main dial is the number of columns per split.
  • Each row is left out by about a third of the trees, which gives a free out-of-bag score.

Try it on your own data: the small model trainer grows a single tree on your CSV and reports its test accuracy: the baseline a random forest (in Python or R) should beat.

New words: random forest, feature subsampling, mtry (columns per split), out-of-bag score, decorrelation.

Check yourself

1A random forest adds what to bagging?
2Adding more and more trees to a random forest
3With 16 columns, the usual default for a classification forest is to try at each split about:

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.