Cross-validation

One train/test split gives you one number, and that number is part luck.

Hold out every slice in turn, score each, average. All your data tests, all of it trains — just never at the same time.

Why one split isn't enough

Reshuffle a single train/test split and the accuracy moves. On small data, a lot.
Enough to make a worse model look better — the exact mistake you measure to avoid.
Stone (1974) and Geisser (1975) turned "hold data back" into a named procedure.
Their answer: don't estimate out-of-sample error from a formula. Measure it.

Cut the data into k folds. Run k rounds. Each round: one fold held out, the rest train. Rotate until every fold has been the test set once.

The estimate is the average of the k fold scores:

\mathrm{CV}_k = \frac{1}{k} \sum_{j=1}^{k} s_j

But report the spread too — how loudly the folds disagreed:

\mathrm{SD}_k = \sqrt{\frac{1}{k} \sum_{j=1}^{k} \left(s_j - \mathrm{CV}_k\right)^2}

$0.75 \pm 0.01$ and $0.75 \pm 0.06$ are not the same result.

Five folds on 768 patients, one frame per round. The held-out fold marches across; the running mean settles; the band is ±1 std.

Fold 4 scores 0.830, fold 2 scores 0.669 — same model, different slice. Mean 0.750, band ±0.057. That width is the honest signal.

The label is 35% positive. Plain contiguous folds swing from 0.255 to 0.416 positive per fold; stratified folds hold ≈ 0.35 everywhere.

Plain ±0.057, stratified ±0.012. Free, and the difference between quotable and noise.

Thirty seeds. Single 80/20 split vs 5-fold CV mean. Same center, a quarter of the spread.

Single split: 0.739 ± 0.040 (0.649–0.799). CV mean: 0.739 ± 0.010 (0.718–0.763).

Any number you'll decide on: cross-validate it. Report mean ± std, always.
Data scarce? That's when CV buys the most. Imbalanced? Stratify — it's free.
Five folds is the default; ten when the model's cheap; leave-one-out only on tiny sets.
The cost is compute, k times over. Tuning on the CV score inflates it — that's what nested CV fixes, and where hyperparameter tuning begins.