Random forests

Week 3's tree, bagged and de-correlated.

Grow many trees on bootstrap samples, split from random feature subsets, and let them vote. The variance that made one tree fragile averages out.

Two cheap tricks

Week 3 warned: never ship one deep tree — it memorizes and falls apart.
Fix: grow a few dozen trees, let them vote. Bagging.
Trick 1 — bootstrap the rows: every tree sees a different resample.
Trick 2 — random feature subset per split: the trees stop thinking alike.
Ho 1995 (random subspaces) + Breiman 1996 (bagging) → Breiman 2001 (random forests).

The data

Two interleaving half-moons — a curved boundary a single tree can only approximate with a staircase.

The math

Average $B$ trees, each variance $\sigma^2$ , pairwise correlation $\rho$ :

\operatorname{Var} = \rho\,\sigma^{2} + \frac{1 - \rho}{B}\,\sigma^{2}

Adding trees kills the second term. Feature subsampling lowers $\rho$ , which lowers the floor $\rho\sigma^2$ . Predict by majority vote:

\hat{y}(x) = \arg\max_{c} \sum_{b} \mathbb{1}\!\left[T_b(x) = c\right]

OOB: ~36.8% of rows sit out each tree — a free held-out estimate.

Watch the boundary smooth

One tree per frame. The jagged single-tree staircase blurs into a smooth consensus; accuracy climbs then plateaus. Variance reduction, drawn in data.

Jagged trees, smooth forest

Three individual trees, each overconfident and wrong in different places. Average them and the jags cancel.

Scratch vs. library

RandomForestClassifier(n_estimators=60, max_depth=8) scores 0.740; our scratch forest 0.758 — within two points. One tree alone: 0.667. Bagging it 60 times bought nine points, no tuning.

Takeaways

A random forest is week 3's tree, bagged and de-correlated. Two lines of randomness.
The reliable default on tabular data: no scaling, mixed types, free OOB score and feature importance, barely any tuning.
More trees never overfit — accuracy plateaus. Stop pruning, start voting.
Where it loses: gradient boosting squeezes out more (next chapters); a forest gives up a tree's readability.