Random forests

What this chapter covers

Week 3 ended on a warning: never ship a single deep tree, because it memorizes the training set and falls apart on new data. This chapter is the fix, and it's almost embarrassingly simple. Take the exact tree you already built, grow a few dozen of them on slightly different views of the data, and let them vote. The variance that made one tree fragile averages out, and you get the model that quietly wins more tabular problems than anything else on a first pass: the random forest.

There are two tricks, both cheap. Each tree trains on a bootstrap sample of the rows — sample the dataset with replacement, so every tree sees a different draw. And at each split, instead of searching all the features, the tree only gets to look at a random handful. The first trick is bagging; the second de-correlates the trees so their mistakes don't line up. We build both from scratch on top of week 3's best_split, then hand the same job to scikit-learn's RandomForestClassifier and watch the numbers agree.

The centerpiece is an animation: we grow the forest one tree at a time and watch the ensemble decision boundary go from a jagged single-tree staircase to a smooth curve that hugs the data, with a side panel tracing accuracy rising and then flattening as the votes pile up. That flattening is the whole idea — variance reduction, drawn in real data.

A bit of history

The single tree was mature by the mid-1980s — CART in 1984, ID3 and C4.5 soon after — and everyone knew its weakness. A tree is a high-variance model: retrain it on a slightly different sample and you can get a completely different tree. The idea that fixed random forests came from two directions that met around the turn of the millennium.

In 1995 Tin Kam Ho, at Bell Labs, published "Random Decision Forests" and then the random subspace method: build each tree on a random subset of the features, so that no two trees are looking at quite the same problem, then combine them. Independently, Leo Breiman — the same Breiman from the CART book — had been working on bagging (1996), short for bootstrap aggregating: train many copies of a model on bootstrap resamples of the data and average them, which he showed drives down the variance of unstable predictors like trees. In 2001 Breiman put the two together in the paper that named the method, "Random Forests." His insight was that bagging alone leaves the trees too similar — they all seize on the same strong feature at the root — so he added Ho's feature subsampling at every split. Bootstrap the rows, subsample the features, average the votes. That recipe, almost unchanged, is what every random forest library ships today.

What we build here is Breiman's forest: week 3's CART, bagged over bootstrap samples and de-correlated by per-split feature subsets. It's worth building once by hand because the same two tricks — resample and average, inject randomness to de-correlate — show up again and again in machine learning.

The intuition

A single tree draws boxes. Its decision boundary is a staircase of horizontal and vertical cuts, and on a curved boundary that staircase is a crude approximation that bends wherever the training points happened to land. Move a few points and the staircase reshuffles. That's the variance problem made visual.

Here's the data we'll use to see it: two interleaving half-moons, a classic toy set with a boundary no straight line and no small set of axis-aligned cuts can follow cleanly.

The two moons curl around each other, and there's noise, so they overlap in the middle. One tree can carve this up, but it'll do it with a blocky staircase that overfits the noise. Now imagine growing a second tree on a different random sample of these points — it draws a slightly different staircase, wrong in different places. A third, wrong somewhere else again. Ask all three where a new point belongs and take the majority answer, and the places where any one tree is wrong get outvoted by the two that happen to be right. Keep adding trees and the jagged individual boundaries blur into a smooth consensus.

That's the forest. No single tree is any better than the trees from week 3 — the magic is entirely in the averaging.

The math

Why does averaging help? Start with the variance. Suppose we have $B$ trees, each an unbiased but noisy predictor of the right answer, and each with variance $\sigma^{2}$ around it. If the trees were independent, the variance of their average would be

More trees, less variance, straight toward zero. That's the dream, and it's why bagging works at all: the individual trees keep their low bias (each still fits the data hard), but averaging washes out the noise that made any one of them unreliable.

The catch is that the trees aren't independent. They're trained on resamples of the same dataset, so they tend to agree — especially at the top, where they all grab the single most predictive feature. Let $\rho$ be the average pairwise correlation between the trees' predictions. Then the variance of the average is

Read the two terms. The second term vanishes as $B \to \infty$ — that's the part bagging kills by adding trees. The first term, $\rho\sigma^{2}$, does not: it's a floor that no number of trees can get you under. If the trees are highly correlated, averaging them barely helps. This is exactly why bagging alone underperforms. The fix is to attack $\rho$ directly, and that's what random feature selection does: by forcing each split to choose from a random subset of features, you stop every tree from anchoring on the same dominant feature, the trees grow into genuinely different shapes, $\rho$ drops, and the floor drops with it. You pay a little bias — each split is made from a worse menu of options — to buy a lot of variance reduction. On most tabular data that trade is heavily in your favor.

Prediction itself is a vote. With $B$ trees, each returning a class $T_b(x)$ for input $x$, the forest predicts the class that the most trees agree on:

where $\mathbb{1}[\cdot]$ is 1 when the tree votes for class $c$ and 0 otherwise. That's the whole aggregation rule — count the votes, take the plurality.

One more number falls out for free. Each bootstrap sample leaves out about $1 - (1 - 1/n)^{n} \to 1 - 1/e \approx 0.368$ of the rows — the out-of-bag set. Every row is out-of-bag for roughly a third of the trees, so we can score each row using only the trees that never trained on it, and average that over all rows. It's a held-out accuracy estimate that costs nothing extra — no separate test split, no cross-validation loop.

What it's good at, what it isn't

A random forest is the most reliable default I know for tabular data. It inherits everything good about trees — no feature scaling, mixed numeric and categorical inputs, nonlinear boundaries, interactions found automatically — and it throws out the one thing that made a single tree unusable, the wild variance. It barely needs tuning: set the number of trees as high as your compute allows, leave the feature subset at the square-root default, and you're most of the way to the best you'll get. It gives you a free OOB accuracy estimate and a feature-importance ranking almost for free. When someone hands me an unfamiliar tabular dataset and asks "is there signal here," a random forest is the first thing I fit, because it's fast to a decent answer and it almost never embarrasses me.

Where does it lose? Two places. It's not the top of the leaderboard: on a dataset you're going to squeeze hard, gradient boosting — the next few chapters — almost always edges it out, because boosting builds trees that correct each other's errors instead of just averaging independent ones. And it gives up the one thing a single tree had going for it, readability. You can print one tree and argue with it; you can't print three hundred. A forest is a black box you trust because it validates well, not because you can read it. If your reason for using trees was interpretability, the forest takes it back.

The data

Two datasets, two jobs, same split as the last two chapters. The concept animation uses the 2-D moons above — 200 points, two interleaving half-moons with noise, generated with a fixed seed and committed as moons.csv. Two dimensions so you can watch the decision boundary directly, and a curved boundary so the jagged-to-smooth story is obvious.

For the honest accuracy number we go back to Pima Indians Diabetes: 768 patients, eight measurements each, a binary diabetes label, the same 70/30 split as weeks 1 and 3 — 537 patients to train on, 231 held out. It's noisy and overlapping, which is exactly where a single tree overfits and a forest earns its keep.

Build it, one function at a time

Almost all of this is week 3. gini, majority, predict — unchanged, so I won't repeat them here; the forest is built out of ordinary CART trees. What changes is small and surgical. Here it is in the order you'd add it.

First, the one line that turns a tree into a forest tree. Week 3's best_split searched every feature; now it searches only a subset the caller hands in:

def best_split(X, y, feature_ids):
    """Search a given subset of features for the split that most reduces Gini.

    This is week 3's best-split search with exactly one change: the outer loop
    runs over `feature_ids` — the random feature subset handed in by the caller
    — instead of every column. For each candidate feature we take the midpoints
    between consecutive sorted unique values as thresholds, send rows with
    feature <= t left and the rest right, and score the split by its impurity
    decrease (parent impurity minus the size-weighted impurity of the two
    children). We return the (feature, threshold, gain) with the largest
    decrease, or None if no split in this subset helps.
    """
    n = X.shape[0]
    parent = gini(y)
    best_gain, best_f, best_t = 0.0, -1, 0.0
    for f in feature_ids:
        values = np.unique(X[:, f])
        thresholds = (values[:-1] + values[1:]) / 2.0
        for t in thresholds:
            left = X[:, f] <= t
            n_left = int(left.sum())
            if n_left == 0 or n_left == n:
                continue
            child = (n_left / n) * gini(y[left]) + \
                    ((n - n_left) / n) * gini(y[~left])
            gain = parent - child
            if gain > best_gain:
                best_gain, best_f, best_t = gain, int(f), float(t)
    if best_f < 0:
        return None
    return best_f, best_t, best_gain

The body is identical to week 3 — same threshold search, same impurity-decrease score. The only change is the outer loop runs over feature_ids instead of range(d). That's the entire de-correlation trick, one substituted iterable.

Now build_tree draws that subset fresh at every node, so each split is made from a different random menu of features:

def build_tree(X, y, max_depth, max_features, rng, depth=0):
    """Grow one CART tree, but pick each split from a fresh random feature
    subset.

    Same recursion as week 3, with the de-correlating twist: at every node we
    draw `max_features` columns at random (without replacement) and only let
    `best_split` look at those. Two trees that saw different bootstrap rows and
    different feature subsets at each node grow into genuinely different shapes,
    which is the whole point — their mistakes won't line up.
    """
    node = {"n": int(len(y)), "gini": gini(y), "prediction": majority(y)}
    if depth >= max_depth or node["gini"] == 0.0:
        node["leaf"] = True
        return node
    d = X.shape[1]
    k = min(max_features, d)
    feature_ids = rng.choice(d, size=k, replace=False)
    split = best_split(X, y, feature_ids)
    if split is None:
        node["leaf"] = True
        return node
    f, t, gain = split
    left = X[:, f] <= t
    node.update({
        "leaf": False, "feature": int(f), "threshold": t, "gain": gain,
        "left": build_tree(X[left], y[left], max_depth, max_features, rng, depth + 1),
        "right": build_tree(X[~left], y[~left], max_depth, max_features, rng, depth + 1),
    })
    return node

Same recursion as week 3, with two lines added: pick k random columns without replacement, and pass them to best_split. A tree grown this way is a little worse on its own — it's sometimes forced to split on a mediocre feature because the good one wasn't in the draw — but that's the price of de-correlation, and the average will more than pay it back.

Next, the bootstrap. Sample n row indices with replacement to make one tree's training bag, and record which rows got left out:

def bootstrap_sample(n, rng):
    """Draw n row indices with replacement (the bag), and return the indices
    left out (the out-of-bag rows).

    About a third of the rows never make it into any given bag — those are the
    tree's private test set. `1 - (1 - 1/n)^n -> 1 - 1/e ~= 0.632` of the rows
    are in-bag, so ~36.8% are OOB. That free held-out set is what makes OOB
    accuracy possible.
    """
    in_bag = rng.integers(0, n, size=n)
    mask = np.zeros(n, dtype=bool)
    mask[in_bag] = True
    oob = np.where(~mask)[0]
    return in_bag, oob

The out-of-bag indices are the ~36.8% of rows this tree never saw. We'll use them in a moment for the free accuracy estimate. This is the other source of randomness: every tree trains on a different bootstrap draw.

With those two pieces, the forest is just a loop. Grow n_trees trees, each on its own bootstrap sample, each split from a random feature subset, and keep every tree's OOB set alongside it:

def build_forest(X, y, n_trees, max_depth, max_features, seed=0):
    """Grow a forest: n_trees CART trees, each on its own bootstrap sample,
    each split drawn from a random feature subset.

    We return the trees alongside each tree's OOB indices so we can score the
    forest on rows every tree was never trained on. One rng, seeded once,
    drives every random choice — bootstrap rows and feature subsets — so the
    whole forest is reproducible.
    """
    rng = np.random.default_rng(seed)
    n = len(y)
    trees, oob_sets = [], []
    for _ in range(n_trees):
        bag, oob = bootstrap_sample(n, rng)
        tree = build_tree(X[bag], y[bag], max_depth, max_features, rng)
        trees.append(tree)
        oob_sets.append(oob)
    return trees, oob_sets

One rng, seeded once, drives every bootstrap draw and every feature subset, so the whole forest is reproducible — rerun it and you get the same trees. There's no fitting beyond growing trees; a random forest has no gradient, no iteration to convergence, nothing to tune mid-training. You grow the trees and you're done.

Prediction is the vote. Every tree predicts every row; the class with the most votes wins:

def forest_vote(trees, X, n_classes):
    """Predict X by majority vote across every tree.

    Each tree casts one vote per row; the class with the most votes wins. This
    averaging is where the variance goes: a single tree is noisy, but the
    consensus of many de-correlated trees is stable.
    """
    votes = np.zeros((X.shape[0], n_classes), dtype=int)
    for tree in trees:
        preds = predict(tree, X)
        votes[np.arange(X.shape[0]), preds] += 1
    return votes.argmax(axis=1)

And the payoff of having kept the OOB sets — a held-out accuracy estimate with no held-out set. For each row, tally votes only from the trees that never trained on it, then compare the majority to the truth:

def oob_score(trees, oob_sets, X, y, n_classes):
    """Out-of-bag accuracy: for each row, tally votes only from the trees that
    never saw it, then compare the majority vote to the truth.

    This is a held-out accuracy estimate that costs nothing — no separate test
    split, no cross-validation. Every row is scored by the subset of trees for
    which it was out-of-bag.
    """
    votes = np.zeros((len(y), n_classes), dtype=int)
    for tree, oob in zip(trees, oob_sets):
        if len(oob) == 0:
            continue
        preds = predict(tree, X[oob])
        votes[oob, preds] += 1
    scored = votes.sum(axis=1) > 0
    pred = votes.argmax(axis=1)
    return float((pred[scored] == y[scored]).mean())

That's the whole model. Two tricks bolted onto week 3's tree — resample the rows, subsample the features — plus a vote and a free scorecard.

Watch it work

Here's the animation the whole chapter builds to. This is the real forest from build_forest growing on the 2-D moons, one tree added per frame. The left panel shades the plane by the ensemble's majority vote — cyan for class 0, purple for class 1 — with the opacity tracking how confident the vote is, so a region where all the trees agree is solid and a contested region near the boundary is faint. The right panel traces two accuracies as the forest grows: test accuracy on held-out points, and the free OOB estimate.

Press play and watch the boundary. The first frame is one tree — a blocky staircase, all hard edges, exactly the kind of overfit boundary week 3 warned about, and its test accuracy is 0.750. As trees pile on, two things happen at once. The hard edges soften: where one tree drew a sharp corner, the next few trees drew it slightly differently, and the vote smears those disagreements into a smooth curve that follows the moons. And the faint contested band along the boundary — the region where the trees split their votes — narrows as the consensus firms up. By thirty trees the boundary is a clean arc through the gap between the moons, test accuracy has settled around 0.883, and OOB tracks just below it at 0.850. The accuracy curve climbs fast for the first handful of trees, bounces a little on the small test set, then flattens — that plateau is the variance-reduction term $\tfrac{1-\rho}{B}\sigma^{2}$ going to zero. Past a couple dozen trees, more trees buy you almost nothing. Reset and run it again; it's deterministic.

To see the de-correlation directly, here are three of the individual trees' boundaries next to the full ensemble. Same data, same code, different bootstrap draws and feature subsets:

Look at how different the three single trees are. Each one is a jagged, overconfident carving of the plane, and they're jagged in different places — tree 1's mistakes aren't tree 2's mistakes. That difference is the whole point; it's what $\rho < 1$ looks like. The ensemble in the fourth panel is what you get when you average them: the idiosyncratic jags cancel and a smooth boundary survives. None of the individual trees is any good. Together they're the best model on the page.

The full implementation

The whole thing, no library, top to bottom — week 3's tree functions plus the handful that bag and de-correlate them. This is the file the animation ran:

"""A random forest classifier, built from scratch.

A random forest is week 3's decision tree, bagged and de-correlated. We grow
many trees; each one is trained on a bootstrap sample of the rows (sample n
rows with replacement) and, crucially, each split is chosen from a random
subset of the features instead of all of them. Prediction is a majority vote
across the trees. The rows a tree never saw — its out-of-bag set — give a free
held-out accuracy estimate with no separate test split.

The tree code below is the week-3 CART with one change: `best_split` searches
a caller-supplied subset of features rather than every feature. That single
change is what turns a pile of near-identical trees into a forest whose errors
cancel. Pure NumPy — no ML library anywhere in this file (pandas only loads
the CSV).

Every function appears in the chapter one step at a time (the `# region:`
markers are what the book's include directives pull in).
"""

import numpy as np
import pandas as pd


# region: gini
def gini(y):
    """Gini impurity of a set of labels: 1 - sum_k p_k^2.

    0 when every label is the same (a pure node), climbing toward 1 - 1/K as
    the K classes even out. This is the number a split tries to drive down —
    identical to week 3, because a forest's trees are ordinary CART trees.
    """
    if len(y) == 0:
        return 0.0
    counts = np.bincount(y)
    p = counts / counts.sum()
    return float(1.0 - np.sum(p * p))
# endregion


# region: majority
def majority(y):
    """The label a leaf predicts (and how the forest votes): the most common
    class. Ties break toward the lower class index, so results are
    deterministic given a fixed seed."""
    return int(np.argmax(np.bincount(y)))
# endregion


# region: best_split
def best_split(X, y, feature_ids):
    """Search a given subset of features for the split that most reduces Gini.

    This is week 3's best-split search with exactly one change: the outer loop
    runs over `feature_ids` — the random feature subset handed in by the caller
    — instead of every column. For each candidate feature we take the midpoints
    between consecutive sorted unique values as thresholds, send rows with
    feature <= t left and the rest right, and score the split by its impurity
    decrease (parent impurity minus the size-weighted impurity of the two
    children). We return the (feature, threshold, gain) with the largest
    decrease, or None if no split in this subset helps.
    """
    n = X.shape[0]
    parent = gini(y)
    best_gain, best_f, best_t = 0.0, -1, 0.0
    for f in feature_ids:
        values = np.unique(X[:, f])
        thresholds = (values[:-1] + values[1:]) / 2.0
        for t in thresholds:
            left = X[:, f] <= t
            n_left = int(left.sum())
            if n_left == 0 or n_left == n:
                continue
            child = (n_left / n) * gini(y[left]) + \
                    ((n - n_left) / n) * gini(y[~left])
            gain = parent - child
            if gain > best_gain:
                best_gain, best_f, best_t = gain, int(f), float(t)
    if best_f < 0:
        return None
    return best_f, best_t, best_gain
# endregion


# region: build_tree
def build_tree(X, y, max_depth, max_features, rng, depth=0):
    """Grow one CART tree, but pick each split from a fresh random feature
    subset.

    Same recursion as week 3, with the de-correlating twist: at every node we
    draw `max_features` columns at random (without replacement) and only let
    `best_split` look at those. Two trees that saw different bootstrap rows and
    different feature subsets at each node grow into genuinely different shapes,
    which is the whole point — their mistakes won't line up.
    """
    node = {"n": int(len(y)), "gini": gini(y), "prediction": majority(y)}
    if depth >= max_depth or node["gini"] == 0.0:
        node["leaf"] = True
        return node
    d = X.shape[1]
    k = min(max_features, d)
    feature_ids = rng.choice(d, size=k, replace=False)
    split = best_split(X, y, feature_ids)
    if split is None:
        node["leaf"] = True
        return node
    f, t, gain = split
    left = X[:, f] <= t
    node.update({
        "leaf": False, "feature": int(f), "threshold": t, "gain": gain,
        "left": build_tree(X[left], y[left], max_depth, max_features, rng, depth + 1),
        "right": build_tree(X[~left], y[~left], max_depth, max_features, rng, depth + 1),
    })
    return node
# endregion


# region: predict
def predict_one(node, x):
    """Walk one sample from the root of a single tree to a leaf."""
    while not node["leaf"]:
        node = node["left"] if x[node["feature"]] <= node["threshold"] \
            else node["right"]
    return node["prediction"]


def predict(tree, X):
    """Predict every row of X with one tree."""
    return np.array([predict_one(tree, x) for x in X])
# endregion


# region: bootstrap
def bootstrap_sample(n, rng):
    """Draw n row indices with replacement (the bag), and return the indices
    left out (the out-of-bag rows).

    About a third of the rows never make it into any given bag — those are the
    tree's private test set. `1 - (1 - 1/n)^n -> 1 - 1/e ~= 0.632` of the rows
    are in-bag, so ~36.8% are OOB. That free held-out set is what makes OOB
    accuracy possible.
    """
    in_bag = rng.integers(0, n, size=n)
    mask = np.zeros(n, dtype=bool)
    mask[in_bag] = True
    oob = np.where(~mask)[0]
    return in_bag, oob
# endregion


# region: build_forest
def build_forest(X, y, n_trees, max_depth, max_features, seed=0):
    """Grow a forest: n_trees CART trees, each on its own bootstrap sample,
    each split drawn from a random feature subset.

    We return the trees alongside each tree's OOB indices so we can score the
    forest on rows every tree was never trained on. One rng, seeded once,
    drives every random choice — bootstrap rows and feature subsets — so the
    whole forest is reproducible.
    """
    rng = np.random.default_rng(seed)
    n = len(y)
    trees, oob_sets = [], []
    for _ in range(n_trees):
        bag, oob = bootstrap_sample(n, rng)
        tree = build_tree(X[bag], y[bag], max_depth, max_features, rng)
        trees.append(tree)
        oob_sets.append(oob)
    return trees, oob_sets
# endregion


# region: forest_vote
def forest_vote(trees, X, n_classes):
    """Predict X by majority vote across every tree.

    Each tree casts one vote per row; the class with the most votes wins. This
    averaging is where the variance goes: a single tree is noisy, but the
    consensus of many de-correlated trees is stable.
    """
    votes = np.zeros((X.shape[0], n_classes), dtype=int)
    for tree in trees:
        preds = predict(tree, X)
        votes[np.arange(X.shape[0]), preds] += 1
    return votes.argmax(axis=1)
# endregion


# region: oob_score
def oob_score(trees, oob_sets, X, y, n_classes):
    """Out-of-bag accuracy: for each row, tally votes only from the trees that
    never saw it, then compare the majority vote to the truth.

    This is a held-out accuracy estimate that costs nothing — no separate test
    split, no cross-validation. Every row is scored by the subset of trees for
    which it was out-of-bag.
    """
    votes = np.zeros((len(y), n_classes), dtype=int)
    for tree, oob in zip(trees, oob_sets):
        if len(oob) == 0:
            continue
        preds = predict(tree, X[oob])
        votes[oob, preds] += 1
    scored = votes.sum(axis=1) > 0
    pred = votes.argmax(axis=1)
    return float((pred[scored] == y[scored]).mean())
# endregion


def accuracy(trees, X, y, n_classes):
    """Fraction of rows the forest's majority vote classifies correctly."""
    return float((forest_vote(trees, X, n_classes) == y).mean())


def n_features_sqrt(d):
    """The classic feature-subset size: floor(sqrt(d)), at least 1. Two
    features (moons) -> 1; eight features (diabetes) -> 2."""
    return max(1, int(np.sqrt(d)))


def load_moons(path="../data/moons.csv"):
    """The 2-D toy set: two interleaving half-moons, a curved boundary a single
    axis-aligned tree can only approximate with a staircase."""
    df = pd.read_csv(path)
    X = df[["x1", "x2"]].to_numpy(float)
    y = df["label"].to_numpy(int)
    return X, y, df


def load_diabetes(path="../data/diabetes.csv"):
    """Pima Indians Diabetes: 768 patients, 8 features, binary Outcome — the
    real dataset for the honest accuracy number, same file as weeks 1 and 3."""
    df = pd.read_csv(path)
    features = [c for c in df.columns if c != "Outcome"]
    X = df[features].to_numpy(float)
    y = df["Outcome"].to_numpy(int)
    return X, y, features

The library version

Nobody hand-rolls a forest in production. scikit-learn's RandomForestClassifier is the same algorithm — bootstrap samples, per-split feature subsets, majority vote — written in C and parallelized across cores:

def sk_forest(X_train, y_train, X_test, y_test,
              n_trees, max_depth, seed=0):
    """Fit a random forest and return (train_acc, test_acc, oob_acc, clf).

    `max_features="sqrt"` is the de-correlating feature subset — the same
    floor(sqrt(d)) our scratch forest uses. `bootstrap=True` and majority vote
    are defaults, so this is the library twin of build_forest + forest_vote.
    `oob_score=True` asks sklearn for the same out-of-bag accuracy we compute
    by hand.
    """
    clf = RandomForestClassifier(
        n_estimators=n_trees, max_depth=max_depth,
        max_features="sqrt", bootstrap=True, oob_score=True,
        random_state=seed, n_jobs=-1,
    )
    clf.fit(X_train, y_train)
    train_acc = float(clf.score(X_train, y_train))
    test_acc = float(clf.score(X_test, y_test))
    oob_acc = float(clf.oob_score_)
    return train_acc, test_acc, oob_acc, clf

max_features="sqrt" is the same $\lfloor\sqrt{d}\rfloor$ feature subset our scratch forest draws, bootstrap=True and the majority vote are defaults, and oob_score=True asks sklearn for the same out-of-bag estimate we compute by hand. The differences are speed and knobs, not the idea: sklearn parallelizes the trees across n_jobs, exposes min_samples_leaf and friends for controlling tree size, and hands back feature_importances_ accumulated across the forest. Same forest, more machinery.

Scratch versus library

Fit both forests — 60 trees, max_depth=8, square-root feature subsets — on the 537-patient training half of Pima, score on the 231 held out. And to make the point about bagging, put a single tree from week 3 next to them:

The two forests land together: our scratch forest scores 0.758 on the held-out test set, sklearn's scores 0.740. They differ by less than two points, which is what you want — different random draws grow different trees, but the ensembles agree, because the whole method is designed to be insensitive to any one tree. Our OOB estimate, 0.767, comes in a hair above the test number and needed no held-out data at all — that's the free scorecard doing its job.

Now the number that matters. The single tree — depth 8, all features, exactly the week-3 model — scores 0.667 on this test set. Bagging that same tree sixty times took it from 0.667 to 0.758, nine points, with no new algorithm and no tuning. That's the entire pitch for random forests in one comparison: same tree, piled up, de-correlated, and voted. For contrast, week 3's carefully depth-limited tree scored 0.736 by capping depth to fight overfitting; the forest beats it while growing its trees to depth 8 and letting them overfit individually, because averaging cleans up the overfitting for free. You stop pruning and start voting.

Here's the OOB and test accuracy climbing as we add trees on the real data, which is the plateau from the animation, drawn on Pima:

Both curves start rough — one tree is noisy — climb steeply through the first ten or so trees, then flatten. That flattening is the whole method telling you when to stop: past a few dozen trees the variance term is spent and you're paying compute for nothing. OOB (orange) tracks test accuracy (purple) closely the whole way, which is why in practice you trust it and skip the separate validation split.

Takeaways

A random forest is week 3's tree, bagged and de-correlated. That's the whole thing. You already had the atom — a CART tree with Gini splits — and this chapter added two lines of randomness on top: bootstrap the rows so every tree sees a different sample, and subsample the features at each split so the trees don't all think alike. Average their votes and the variance that made a single tree unusable washes out. No gradients, no iteration, barely any tuning.

Reach for it as your default on tabular data. When you get a new dataset and want to know whether there's signal, fit a random forest first: it handles mixed feature types without scaling, finds interactions on its own, gives you an OOB accuracy estimate without a held-out set, and ranks your features by importance, all in one fit and almost no configuration. Set the tree count as high as your compute allows — accuracy plateaus, it doesn't overfit with more trees — leave the feature subset at the square-root default, and move on. It's the honest baseline every other model has to beat.

Know where it stops, too. If you're going to squeeze a dataset for every last point, gradient boosting usually wins, because it grows trees that fix each other's mistakes instead of averaging independent ones — that's the next stretch of the course, AdaBoost and then gradient boosting and XGBoost. And if you needed a tree because a human had to read it, the forest takes that back; three hundred trees are a black box. But for the wide middle — most tabular problems, most of the time, when you want a strong answer fast and can't afford to babysit a model — the random forest is the one I reach for first, and it's usually still standing at the end.