Tuning hyperparameters

Some numbers a model learns from data. Some you have to hand it first — depth, neighbors, C, gamma. Those are hyperparameters.

Tuning is the honest loop around cross-validation: try a configuration, score it with k-fold, keep the winner — without fooling yourself.

Grid folklore, then Bergstra

Grid search has no inventor. It's the reflex: list values, try every combo, keep the best.
Trivial to write, trivial to parallelize, deterministic. Also quietly wasteful.
Bergstra & Bengio (2012): for the same budget, random search usually beats grid — and the gap grows with dimensions.
Why: most knobs don't matter. A grid re-tests the useless ones; random varies the important ones every trial.

Two knobs, C and gamma, five values each. Every cell is a 5-fold CV accuracy. The search discovers this one cell at a time — it can't see it first.

Bright ridge in the middle. Brightest cell: C=10, gamma=0.01.

Cross-validated score of a configuration $\lambda$ :

\mathrm{CV}(\lambda) = \frac{1}{k} \sum_{j=1}^{k} \frac{1}{|F_j|} \sum_{i \in F_j} \mathbb{1}\!\left[\, \hat{f}_{\lambda}^{(-j)}(x_i) = y_i \,\right]

Tuning is its argmax over the search set:

\lambda^{*} = \arg\max_{\lambda \in \Lambda} \; \mathrm{CV}(\lambda)

Grid: $\Lambda$ is the lattice. Random: $\lambda_t \sim p(\lambda)$ , log-uniform for scale knobs.

Grid crawls all 25 cells (star = best so far). Random scatters the same budget — 12 draws — and lands near the peak in a fraction of the fits.

Grid's best: 0.781 in 25 fits. Random: 0.778 by draw 4, done in 12. On two knobs, modest. On ten, a chasm.

Fix C, sweep gamma. Cyan = held-out CV, orange = training folds. The peak is the sweet spot; the gap is the overfitting.

Small gamma: both low, high bias. Large gamma: training → 1.0, validation collapses. Peak ≈ 0.003.

Tune on 576 training patients with inner CV. Report on 192 held-out patients, touched once.

CV: 0.774 → 0.781, robust. Test: 0.734 → 0.776 (192 points — trust the CV, glance at the test).

Start with random search. Grid only wins on 1–2 knobs; past that it wastes budget on knobs that don't matter.
Never tune on the test set — structurally. The score you optimize is contaminated the instant you optimize it.
Log-scale the search for C, gamma, learning rate, regularization. Sample the exponent.
Diminishing returns: tuning was a polish (+0.7 pt CV). A better model, feature, or cleaner data beats perfect tuning. Next: Bayesian optimization (Optuna, Hyperopt) — adaptive, not blind, but the same objective.