Ridge regression

Linear regression with a leash: minimize squared error plus a penalty on the size of the weights.

Tames correlated features and overfit — keeps every feature, just small.

One term changes everything

• OLS minimizes squared error. Ridge minimizes squared error plus $\lambda \lVert w \rVert_2^2$.
• Closed form is the normal equation plus a diagonal: $w = (X^{\top}X + \lambda I)^{-1} X^{\top} y$.
• The intercept is never penalized (0 in the $\lambda I$ slot).
• $\lambda I$ also makes $X^{\top}X$ invertible even when features are collinear.

Why bother: correlated features

s1 and s2 in the diabetes data are correlated 0.90. OLS trades wild offsetting weights between them; ridge charges rent and picks small, stable ones.

The coefficient path

Every frame is one real refit. Turn lambda up; every weight shrinks toward zero, the correlated ones fastest. The bottom panel finds the sweet spot.

Validation MSE bottoms out at lambda ≈ 31.6 — a real U, not "more is better."

L2 shrinks, it doesn't select

Every ridge coefficient is shorter than its OLS twin; none are zeroed. The weight-vector norm drops from 45.5 to 40.6.

Scratch vs. library

Our closed-form ridge and sklearn's Ridge agree to ~1e-13. Ridge beats unregularized OLS on held-out data.

Test R²: OLS 0.339 → ridge 0.363. Same model class, better generalization.

Takeaways

• Reach for ridge when coefficients are huge, flip signs, or test error trails train error.
• Standardize first — the penalty compares coefficients directly.
• One knob (lambda); pick it by a validation sweep or RidgeCV.
• L2 shrinks all coefficients toward zero but keeps them. Want selection? That's lasso (L1). Both? Elastic net.