Train, validation, and test splits

What this chapter covers

There is one mistake that shows up in more broken machine-learning projects than any other, and it has nothing to do with the model. It's scoring the model on the same data it learned from, then believing the number. That number is always good. It's good the way a student who saw the exam beforehand scores well — the score measures memory, not learning. This whole chapter is about the fix, which is boring, mechanical, and non-negotiable: hold some data out, train without it, and score on it once.

We build the splits by hand in NumPy — a random train/test split, a three-way train/validation/test split, and a stratified split that keeps the class proportions intact — then hand the same job to scikit-learn and confirm the sizes and proportions line up. The centerpiece is an animation you should not skip: a decision tree grown deeper and deeper, with its training accuracy and its held-out test accuracy plotted side by side, frame by frame. You watch the training line march to a perfect 1.0 while the test line peaks early and then turns and falls. The gap between those two lines is the model memorizing, drawn in real data. That gap is the entire argument for holding data out, and once you've seen it open you won't forget to split again.

Back in week 1 the threshold classifier already whispered this — its accuracy on the training data ran optimistic, and the honest face-off used a held-out split. This chapter is that footnote promoted to the main text.

A bit of history

The idea that you should test a model on data it didn't see is older than machine learning, and it started as a warning about a specific lie. In 1931 Selig Larson, working in educational psychology, noticed that regression weights fit on one sample and then applied to a fresh sample predicted noticeably worse than they had on the sample they were fit on. The correlation "shrank." He'd found overfitting in linear regression thirty years before anyone had a computer to overfit with, and the fix he pointed at was to fit on one batch of data and check on another.

Pattern recognition made it a formal experimental protocol. In 1962 Wallace Highleyman, at Bell Labs, wrote down how to design a recognition experiment properly, splitting the examples into a design set and a test set — the "Highleyman split" — so that the reported error rate meant something. And in 1974 Mervyn Stone gave the whole practice its statistical spine in a paper titled "Cross-validatory choice and assessment of statistical predictions," formalizing what it means to hold data out, rotate which data you hold out, and use the result both to choose a model and to assess it. Stone's paper is why we have a clean vocabulary for it at all; leave-one-out, the extreme case, had already shown up in Lachenbruch and Mickey's work in 1968. Cross-validation is next chapter's subject — this chapter is the single-split version it generalizes, and it's worth getting exactly right first.

The intuition

You have a fixed pile of labeled data and two things you want from it that fight each other. You want to use as much of it as possible to train, because more data makes a better model. And you want an honest estimate of how the model will do on data it's never seen, which you can only get by keeping some data away from training entirely. Every row you hold out is a row you didn't train on. That's the tension, and the split is how you resolve it.

The plainest version is two piles. Shuffle the rows, cut off a chunk — say 30% — and call it the test set. Train on the other 70%. Score on the 30%. That score is your estimate of real-world performance, and it's honest exactly because the model never touched those rows while learning.

Once you start tuning, though, two piles aren't enough, and this is the part people get wrong. The moment you look at the test score, try a deeper tree, look again, and keep the version that scored better, you've started training on the test set — slowly, through your own decisions. So you cut a third pile. The validation set is the one you're allowed to peek at as often as you like while you pick models and settings. The test set you touch once, at the very end, to report a single honest number. Here's the three-way split on our data, drawn to scale:

A 60/20/20 cut of 768 patients: 460 to train, 154 to tune on, 154 held back for the final verdict. The exact sizes come from results.json. Nothing about those fractions is sacred — 80/10/10 is common when data is scarce, 60/20/20 when it's plentiful — but the shape is always the same: a big pile to learn from, a middle pile to choose with, a last pile you don't spend until the end.

The math

Write the model as a function $h$ that maps a patient's measurements $x$ to a predicted label. What you actually care about is the generalization error: how often $h$ is wrong on a brand-new patient drawn from the same population $\mathcal{D}$ the data came from.

You can never compute that expectation — it's over every patient who could ever exist, and you have 768 of them. So you estimate it. The wrong estimate is the error on the same rows you trained on:

This one is biased low, and not by a little. The model chose its parameters to minimize exactly this quantity, so of course it looks good here — it optimized for it. The right estimate uses rows the model never saw while training:

Because those rows were held out, this average is an unbiased estimate of the generalization error — the number you'd actually see in production. The split itself is just a partition of the $n$ row indices into disjoint pieces that cover everything:

with the test size fixed by the fraction you chose, rounded up so the held-out set is never empty:

Stratification adds one constraint on top of the partition. If class $k$ is a fraction $n_k/n$ of the full data, a stratified split makes every piece carry that same fraction:

A blind random split gets this right on average, but any single draw wobbles — and the smaller the test set or the rarer the class, the more it wobbles. Pinning it exactly is what stratification buys you.

What it costs, what it buys

Holding data out is not free, and it's worth being honest about the price. Every row in the test set is a row your model didn't get to learn from, so a hold-out split makes your model slightly worse than one trained on all the data — you're spending accuracy to buy an honest measurement. And the measurement itself is noisy: a test set of 154 patients gives an error estimate with real variance, so a difference of a point or two between two models on one test split might be nothing. Cross-validation, next chapter, exists precisely to squeeze a lower-variance estimate out of the same scarce data.

What you buy is the only number that matters: an estimate of how the model performs on data it has never seen, which is the only condition it will ever face in production. Without a hold-out you don't have a bad estimate of that number — you have no estimate at all, just a measurement of how well the model memorized. That trade is not close. Spend the data.

The data

Same set as week 1, so the numbers are comparable across chapters: the Pima Indians Diabetes data, 768 patients, eight measurements each, and a binary label for whether they were diagnosed with diabetes. Of the 768, 268 are positive and 500 negative — a positive rate of 0.349, per results.json. That imbalance matters here: it's exactly the kind of skew a careless random split can distort, and the reason stratification earns its place later in the chapter.

The point of this chapter isn't the model, so the model stays in the background: a decision tree, the same one week 3 builds from scratch, used here only as something with a flexibility knob I can turn. Turn the knob up and the tree gets freer to memorize. That's all I need to make the case for splitting.

Build it, one function at a time

The whole thing is four functions, and none of them is longer than a paragraph. A split is not a hard piece of code — the discipline is in using it, not writing it. Everything starts with a shuffle, because the rows in a real dataset are almost never in random order. They're sorted by date, grouped by hospital, clustered by whatever the collection process imposed, and if you slice the last 30% off an ordered file you get a test set that's systematically different from your training set. So: shuffle first, seeded, always.

def shuffle_indices(n, seed=42):
    """A reproducible permutation of the row positions 0..n-1.

    Every split starts here. Seed it and the split is identical forever; that
    is the whole difference between a result you can rerun and a number someone
    just has to trust.
    """
    rng = np.random.default_rng(seed)
    return rng.permutation(n)

The seed is not a detail. A seeded permutation is the difference between an experiment you can rerun and get the same answer, and a number that changes every time you look at it. Seed everything that touches randomness; it's the cheapest reproducibility you'll ever buy.

With shuffled indices, the plain two-way split is one cut:

def train_test_split(n, test_frac=0.3, seed=42):
    """Shuffle the row positions, then slice the last chunk off as the test set.

    n_test = ceil(n * test_frac), carved from the end so training keeps the
    rest — the same sizes scikit-learn produces for the same fraction. Returns
    (train_idx, test_idx) as index arrays, so the caller can split any column.
    """
    idx = shuffle_indices(n, seed)
    n_test = int(np.ceil(n * test_frac))
    split = n - n_test
    return idx[:split], idx[split:]

I return indices rather than the split data itself, so the same split applies to your features, your labels, and anything else keyed by row. The test size is ceil(n * test_frac) counted from the end — which is exactly how scikit-learn sizes it, so our split and theirs land on the same counts.

The three-way split is the same idea with two cuts instead of one. The test slice comes off the untouched end and gets set aside; validation comes next; the rest trains:

def train_val_test_split(n, val_frac=0.2, test_frac=0.2, seed=42):
    """One shuffle, two cuts. The test slice is carved off first and set aside;
    the validation slice comes next; everything left over trains.

    Test comes off the untouched end on purpose — it is the slice you promise
    not to look at until the very end. Returns (train_idx, val_idx, test_idx).
    """
    idx = shuffle_indices(n, seed)
    n_test = int(np.ceil(n * test_frac))
    n_val = int(np.ceil(n * val_frac))
    test_idx = idx[n - n_test:]
    val_idx = idx[n - n_test - n_val:n - n_test]
    train_idx = idx[:n - n_test - n_val]
    return train_idx, val_idx, test_idx

The ordering is deliberate. Carve the test set off first and leave it alone. Everything you do afterward — every model you compare on the validation set — happens on the other two piles, and the test indices sit there untouched until the last line of the project.

Now the one with teeth. A random split keeps the class balance right on average, but "on average" is cold comfort when you have one dataset and one split. A stratified split guarantees it by splitting each class separately and taking the same fraction from each:

def stratified_split(y, test_frac=0.3, seed=42):
    """A random split done one class at a time, so each class keeps its share.

    For every distinct label, shuffle just that class's rows and send a
    test_frac slice to the test set. The test set then carries the same class
    proportions as the full data — which a blind random split only matches on
    average, not on any single draw. Returns (train_idx, test_idx).
    """
    rng = np.random.default_rng(seed)
    y = np.asarray(y)
    train_idx, test_idx = [], []
    for c in np.unique(y):
        members = np.where(y == c)[0]
        members = members[rng.permutation(len(members))]
        n_test = int(round(len(members) * test_frac))
        test_idx.append(members[:n_test])
        train_idx.append(members[n_test:])
    return np.concatenate(train_idx), np.concatenate(test_idx)

Loop over the classes, shuffle each class's rows on their own, peel off a test_frac slice from each, and glue the pieces back together. The test set ends up with the same proportion of diabetics as the full data, exactly, not approximately. The helper that measures what we're preserving is trivial, and it's what the comparison chart later is built from:

def class_proportions(y):
    """Fraction of rows in each class — the thing a stratified split preserves."""
    y = np.asarray(y)
    classes, counts = np.unique(y, return_counts=True)
    return {int(c): float(n) / len(y) for c, n in zip(classes, counts)}

Watch it work

This is the argument. Everything above is setup for this one animation, so slow down here.

We take a stratified train/test split of the Pima data and fit a decision tree, then fit it again with a deeper depth cap, and again, deeper each time — from a depth-1 stump up to depth 16. A deeper tree is a more flexible model: it can carve the training data into finer and finer boxes until each box holds a handful of points. Each frame is one real tree at one real depth. The cyan line is its accuracy on the training data; the purple line is its accuracy on the held-out test set it never saw. Press play and watch the two lines separate.

Here's what you're looking at, and it's the whole chapter in one picture. At depth 1 the tree is a stump — train accuracy 0.740, test accuracy 0.678, the two lines almost touching. As the tree deepens, the cyan training line climbs without pause and reaches a flawless 1.000 by depth 16: given enough depth, the tree memorizes every training patient perfectly. The purple test line does something completely different. It rises to 0.735 at depth 4 and then turns around and falls, all the way back to 0.674 by depth 16. The gap between the two lines opens from 0.062 to 0.326 — the numbers are in the caption and in results.json.

That widening gap is overfitting, and it is not subtle once you can see it. Every level of depth past four makes the tree look better on data it has already seen and worse on data it hasn't. If you only ever looked at the training line — the one you get for free, without holding anything out — you'd conclude that deeper is always better and ship the depth-16 tree that scores a perfect 1.000. It's the worst model on the chart. The only way to know that is the purple line, and the only way to draw the purple line is to have held data out.

Now the same lesson as a single, blunt comparison. Take one overfit tree — an unrestricted one, no depth cap, free to memorize — and score it two ways: on the data it trained on, and on the held-out test set.

On its own training data the tree scores a perfect 1.000. On the held-out test set it scores 0.674. Same model, same day, same code — the only thing that changed is whether the data was seen during training. If you reported the 1.000, you would be off by a third and confidently wrong. This is not a corner case or a pathological model; it's the default outcome of scoring on training data, and it's why "what's your accuracy?" is an incomplete question until you've said accuracy on what.

The full implementation

Four functions and a loader, no library, top to bottom. This is the file the animation actually ran:

"""Honest data splits, built from scratch.

Holding data out is the one habit that separates measuring learning from
measuring memorization. This file builds the splits by hand in NumPy: a random
train/test split, a three-way train/validation/test split, and a stratified
split that preserves class proportions. Pure NumPy — the only model in the whole
chapter lives in library.py, and it's just a prop to make the overfitting gap
visible.

Every function below appears in the chapter one step at a time (the
`# region:` markers are what the book's include directives pull in).
"""

import numpy as np
import pandas as pd


# region: shuffle_indices
def shuffle_indices(n, seed=42):
    """A reproducible permutation of the row positions 0..n-1.

    Every split starts here. Seed it and the split is identical forever; that
    is the whole difference between a result you can rerun and a number someone
    just has to trust.
    """
    rng = np.random.default_rng(seed)
    return rng.permutation(n)
# endregion


# region: train_test_split
def train_test_split(n, test_frac=0.3, seed=42):
    """Shuffle the row positions, then slice the last chunk off as the test set.

    n_test = ceil(n * test_frac), carved from the end so training keeps the
    rest — the same sizes scikit-learn produces for the same fraction. Returns
    (train_idx, test_idx) as index arrays, so the caller can split any column.
    """
    idx = shuffle_indices(n, seed)
    n_test = int(np.ceil(n * test_frac))
    split = n - n_test
    return idx[:split], idx[split:]
# endregion


# region: train_val_test_split
def train_val_test_split(n, val_frac=0.2, test_frac=0.2, seed=42):
    """One shuffle, two cuts. The test slice is carved off first and set aside;
    the validation slice comes next; everything left over trains.

    Test comes off the untouched end on purpose — it is the slice you promise
    not to look at until the very end. Returns (train_idx, val_idx, test_idx).
    """
    idx = shuffle_indices(n, seed)
    n_test = int(np.ceil(n * test_frac))
    n_val = int(np.ceil(n * val_frac))
    test_idx = idx[n - n_test:]
    val_idx = idx[n - n_test - n_val:n - n_test]
    train_idx = idx[:n - n_test - n_val]
    return train_idx, val_idx, test_idx
# endregion


# region: stratified_split
def stratified_split(y, test_frac=0.3, seed=42):
    """A random split done one class at a time, so each class keeps its share.

    For every distinct label, shuffle just that class's rows and send a
    test_frac slice to the test set. The test set then carries the same class
    proportions as the full data — which a blind random split only matches on
    average, not on any single draw. Returns (train_idx, test_idx).
    """
    rng = np.random.default_rng(seed)
    y = np.asarray(y)
    train_idx, test_idx = [], []
    for c in np.unique(y):
        members = np.where(y == c)[0]
        members = members[rng.permutation(len(members))]
        n_test = int(round(len(members) * test_frac))
        test_idx.append(members[:n_test])
        train_idx.append(members[n_test:])
    return np.concatenate(train_idx), np.concatenate(test_idx)
# endregion


# region: class_proportions
def class_proportions(y):
    """Fraction of rows in each class — the thing a stratified split preserves."""
    y = np.asarray(y)
    classes, counts = np.unique(y, return_counts=True)
    return {int(c): float(n) / len(y) for c, n in zip(classes, counts)}
# endregion


def load_data(path="../data/diabetes.csv"):
    """Pima Indians Diabetes dataset: 768 patients, 8 features, Outcome."""
    return pd.read_csv(path)

The library version

You will not hand-roll splits in practice, and after building them once you shouldn't. sklearn.model_selection.train_test_split is the function you'll call, and it does exactly what ours does — shuffle, then slice:

def sk_split(X, y, test_size=0.3, seed=42):
    """A plain random split. Returns X_train, X_test, y_train, y_test. The
    sizes match our from-scratch train_test_split for the same fraction and
    seed — both carve ceil(n * test_size) rows into the test set."""
    return sk_split_fn(X, y, test_size=test_size, random_state=seed, shuffle=True)

Same shuffle-and-cut, same test size for a given fraction, seeded by random_state the way ours is seeded by seed. The one addition worth knowing is stratify. Pass the labels to it and you get our stratified split for free:

def sk_stratified_split(X, y, test_size=0.3, seed=42):
    """The same call with stratify=y. Now the test set gets each class in the
    same proportion it holds in the full data — the library twin of our
    stratified_split."""
    return sk_split_fn(X, y, test_size=test_size, random_state=seed,
                       shuffle=True, stratify=y)

That single keyword is the library twin of the whole per-class loop we wrote. In real work stratify=y is the argument I reach for by default on any classification problem — it costs nothing and it removes a way to be quietly wrong. The model in the animation is sklearn's tree too, turned one depth at a time; it's the prop, not the point:

def tree_at_depth(X_train, y_train, X_test, y_test, depth, seed=42):
    """Fit one decision tree capped at max_depth=depth; return (train_acc,
    test_acc). Deepen the cap and you widen the model's freedom to memorize the
    training rows — that is the flexibility dial the overfitting animation
    turns."""
    clf = DecisionTreeClassifier(max_depth=depth, random_state=seed)
    clf.fit(X_train, y_train)
    return float(clf.score(X_train, y_train)), float(clf.score(X_test, y_test))

Scratch versus library

The honest comparison here isn't scratch-versus-library accuracy — both split the same rows into the same sizes, so there's nothing to race. The comparison that matters is random-versus-stratified, and it's about class balance, not accuracy. Here's the positive-class share in the test set three ways: the full dataset, a plain random split, and a stratified split.

The full data is 0.349 positive. The stratified test set is 0.348 — the boundary between the colors sits right where the full data's does, which is the guarantee stratification makes. The random test set drifts to 0.338: close, but off, and it drifted purely by luck of the shuffle. Every number here is from results.json.

On this data the drift is small, and I want to be straight about that rather than oversell it — Pima is only mildly imbalanced and the test set is a healthy 231 rows. But the drift grows fast in exactly the situations that hurt most: a rarer positive class, a smaller test set, a split into many folds. When your positive class is 3% of the data, a random test split can easily land at 1% or 5%, and now your headline metric is measured on a population that doesn't match production. Stratification removes that failure mode for one keyword. There's no reason not to use it, so I always do.

Takeaways

Never score on the training data and believe it. That's the whole chapter, and it sounds too obvious to state until you've watched a colleague — or yourself — post a validation-free accuracy that evaporated in production. The training score measures memorization; the held-out score measures learning; they are different numbers and the animation showed them diverging by a third of the scale. Hold data out, every time, no exceptions.

Keep the two hold-out sets straight, because they do different jobs. The validation set is where you make decisions — which model, which depth, which features — and you can look at it as often as you like. The test set answers one question, once: how good is the final model, honestly? The instant you use the test set to choose something, it stops being a test set and becomes another validation set, and you no longer have an unbiased number. Cut it off first, don't touch it until the end, report it once.

Two habits that come nearly free. Seed every shuffle, so your split is the same tomorrow as it was today and your results are reproducible. And stratify your splits on the label whenever you're classifying, so a rare class doesn't get lopsided by luck — it costs one keyword and closes a whole category of quiet bugs. Watch for leakage while you're at it: scaling or imputing before you split, so test-set statistics bleed into training; or splitting rows that belong to the same patient across train and test. Both inflate your score the same optimistic way training-set scoring does, and both are harder to spot because the split is technically there.

A single split is the honest minimum. Its weakness is variance — one test set of 154 rows gives a wobbly estimate, and two models a point apart might be tied. The fix is to rotate which data you hold out and average, which is cross-validation, the next chapter. But cross-validation is this idea repeated, not a replacement for it. Get the single split right, keep the test set sacred, and everything downstream has a foundation to stand on.