Elastic net

Linear regression with both penalties at once: lasso's L1 for sparsity, ridge's L2 for stability.

One knob, the mix ratio, slides from ridge to lasso. The default when you have many correlated features and want sparsity without lasso picking favorites.

The problem: correlated features

s1 and s2 — cholesterol and LDL — correlate at 0.897. Lasso keeps one and drops the other at random; ridge keeps both but never zeros anything.

The objective

L(w, b) = \frac{1}{2n}\sum_{i}\left(\hat{y}_i - y_i\right)^2 + \lambda\left[\alpha\lVert w\rVert_1 + \frac{1-\alpha}{2}\lVert w\rVert_2^2\right]

$\lambda$ — penalty strength. $\alpha$ — the L1/L2 mix.
$\alpha = 1$ → pure lasso. $\alpha = 0$ → pure ridge.
Elastic net is the convex combination; those are its endpoints.

The update: soft-threshold over a shrink factor

w_j \leftarrow \frac{S\!\left(\rho_j,\; \lambda\alpha\right)}{1 + \lambda(1-\alpha)}, \qquad S(z, \gamma) = \operatorname{sign}(z)\,\max(|z| - \gamma, 0)

Numerator: the L1 part — soft-threshold, can snap a coefficient to exactly zero.
Denominator: the L2 part — a constant shrink that ties correlated features together.
Cycle it over the coefficients (coordinate descent) until they stop moving.

Watch it: sweep ridge → lasso

Fix $\lambda = 1$ , sweep $\alpha$ from 0 (ridge) to 1 (lasso), refitting from scratch every frame. Watch the dense ridge fit go sparse.

Ten coefficients nonzero under ridge; seven under lasso. Watch s2 vanish at the far right — that's lasso breaking the correlated group.

The grouping effect

Same $\lambda$ , three regimes. Ridge and elastic net keep the s1/s2 pair together; lasso pours everything into s1 (-4.84) and cuts s2 to zero.

Scratch vs. library

Coordinate descent vs. sklearn.linear_model.ElasticNet, same $\lambda$ and $\alpha$ , held-out diabetes patients.

Test R² 0.3653 for both — coefficients agree to fourteen decimals. One convex minimizer, every correct solver finds it.

Takeaways

The default for many correlated features: sparsity without lasso's instability.
The L2 part ties correlated coefficients together (the grouping effect); the L1 part still zeros the dead weight.
On this data the penalty didn't buy accuracy (R² ≈ 0.37 either way) — it bought a stable, readable coefficient vector.
It's not a third model. It's the dial between ridge and lasso; the right setting is a cross-validation search, not a guess.