Train, validation, and test splits

Score a model on the data it trained on and the number is always good — it measures memory, not learning.

Hold data out, train without it, score on it once.

The one mistake

Training accuracy is biased low on error, high on accuracy — the model optimized for it.
The only honest number comes from rows the model never saw while training.
Two piles: train on 70%, score on the held-out 30%.
Three piles once you tune: validation to choose, test to report — touched once, at the end.

The split, to scale

60/20/20 of 768 patients: 460 train, 154 validation, 154 test.

The generalization gap

\hat{\varepsilon}_{\text{train}} \le \varepsilon(h) \approx \hat{\varepsilon}_{\text{test}}

Training error is biased low because the model minimized exactly it. Test error, on held-out rows, is the unbiased estimate of the error you'll see in production.

Watch the gap open

A decision tree grown deeper each frame. Cyan = train accuracy, purple = held-out test. Train marches to 1.000; test peaks at depth 4 then falls.

The gap opens from 0.062 to 0.326. That gap is the model memorizing.

Optimistic vs honest

One unrestricted tree, scored two ways. On its training data: 1.000. On the held-out test set: 0.674. Same model — the only change is whether the data was seen.

Stratify for imbalance

Positive-class share in the test set: full data 0.349, stratified 0.348, random drifts to 0.338 — and the drift grows with rarer classes and smaller sets.

Takeaways

Never score on training data and believe it — memorization, not learning.
Validation to choose, test to report once. Touch the test set and it stops being one.
Seed every shuffle; stratify on the label for classification. Both nearly free.
Watch for leakage: scale/impute after the split, keep same-entity rows on one side.
A single split is the honest minimum; rotating it and averaging is cross-validation — next.