Overfitting and the bias-variance tradeoff

Every model has a flexibility knob. Turn it up and training error keeps falling — but test error falls, bottoms out, and climbs back. That U is the most important picture in supervised learning.

The three fits, one dataset

A straight line can't bend (bias). A degree-11 curve chases the noise (variance). Degree 5 finds the sine.

The decomposition

Expected squared error splits into three pieces:

• bias² — how far the average fit sits from the truth (too stiff).
• variance — how much a single fit jitters as the data changes (too flexible).
• noise $\sigma^2$ — the irreducible floor no model beats.

Total error is their sum. The sweet spot minimizes the sum, not either alone.

Watch the U form

One real refit per frame, degree 1 → 11. Fit tightens up top; test error carves its U down below.

Train MSE slides to 0.038; test MSE bottoms at 0.061 (degree 5) then climbs to 0.817 (degree 11) — target units².

Bias falls, variance rises, they cross

Bias² 0.123 → 0.003. Variance 0.012 → 1.602. They cross near degree 4; total error is the U.

The cost of the wrong degree

Degree 11 scores 0.817 vs. the sweet spot's 0.061 — 13× the error — yet looks better on training data. Judge on held-out data or ship the worse model.

Takeaways

• The U-curve is universal: tree depth, network width, k in k-NN, training steps.
• Bias-limited (bad on both)? More flexibility or better features.
• Variance-limited (great on train, poor on test)? Simplify, regularize, or more data.
• Regularization (ridge/lasso) makes the trade continuous; cross-validation finds the bottom without burning a held-out set.
• Double descent complicates the classic U for very large models — worth knowing, but the single U is the map for everything in this course.