Lasso regression

Linear regression with an L1 penalty. It shrinks coefficients and sets some to exactly zero — the model selects its own features.

Ridge shrinks smoothly and never reaches zero. Lasso snaps coefficients onto zero, one at a time.

One term changes everything

L(w, b) = \frac{1}{2n} \sum_{i} \left( y_i - w^{\top} x_i - b \right)^2 + \lambda \sum_{j} \lvert w_j \rvert

The L1 penalty charges a flat toll per unit of weight — the pull toward zero never fades.
Ridge uses $\lambda \sum_j w_j^2$ instead; that's the whole behavioral difference.
$\lambda = 0$ is ordinary least squares; crank it up and coefficients die one by one.

The soft-threshold does the work

No normal equation — the absolute value isn't differentiable at zero. Fit one coordinate at a time instead:

w_j = S_{\lambda}(\rho_j), \qquad S_{\lambda}(z) = \operatorname{sign}(z)\,\max(\lvert z \rvert - \lambda,\, 0)

Shrink toward zero by $\lambda$ ; if that pushes past zero, stop at zero. That clip is the snap. Cycle the coordinates until nothing moves.

Sparse recovery

Twelve features, only four real. OLS smears weight across the noise; lasso zeroes the eight it should and keeps the four it should.

Watch the coefficients snap to zero

Fifty real refits, $\lambda$ small to large. Each bar shrinks; one by one a coefficient hits zero and goes gray. The bottom panel counts the survivors.

On diabetes, body mass index and s5 hang on longest — they're what the model leans on hardest.

Pick lambda on held-out data

Minimum at $\lambda \approx 2.21$ : lasso keeps 7 of 10 features, drops age, s1, s4.

Scratch vs. library

Our lasso and sklearn agree to eight decimals (R² 0.362). Seven features beat full OLS's ten (R² 0.339).

Takeaways

Reach for lasso when you have more features than you trust — it shrinks and selects in one convex fit.
The zeros are the payoff; ridge can't give you them.
Trust the sparsity, not the exact selection — correlated features make the choice unstable.
Standardize first: one $\lambda$ penalizes every coefficient equally only if they share a scale.
The soft-threshold you built is the piece the elastic net (next chapter) is built on.