Elastic net

What this chapter covers

Elastic net is linear regression with two penalties bolted on at once: the L1 penalty from lasso, which drives coefficients to exactly zero and gives you a sparse model, and the L2 penalty from ridge, which keeps coefficients small and well-behaved when your features move together. A single knob, the mix ratio, slides between them. Turn it all the way one way and you have lasso; all the way the other and you have ridge; anywhere in the middle and you get both effects at once.

That "both at once" is the whole reason it exists. Lasso's sparsity is wonderful right up until your features are correlated, at which point lasso gets twitchy — it grabs one feature out of a correlated group, zeroes the rest, and which one it grabbed can flip if you resample the data. Ridge is stable in exactly that situation but never gives you a zero, so you keep all your features. Elastic net is the default I reach for when I have a pile of correlated features and I want sparsity without lasso picking favorites at random. It builds directly on the linear regression chapter and sits between the ridge and lasso chapters as their synthesis; if you've read those, this is the knob that connects them.

We build it from scratch in NumPy — the elastic-net objective, coordinate descent with a soft-threshold for the L1 part and a shrink factor for the L2 part, predict, R² — and check it against sklearn.linear_model.ElasticNet. The real data is the diabetes dataset: 442 patients, ten standardized features, and a serum-cholesterol group that's correlated enough to make the grouping effect show up in plain sight.

A bit of history

The two penalties came first, decades apart. Ridge regression is the older one: Arthur Hoerl and Robert Kennard published it in 1970 as a fix for multicollinearity, adding a squared-coefficient penalty so that $X^{\top}X$ became invertible again and the wild coefficient swings that correlated features produce got tamped down. It shrinks, but it never selects — every coefficient stays in the model, just smaller. Then in 1996 Robert Tibshirani published the lasso, swapping the squared penalty for an absolute-value one, and that single change bought sparsity: the absolute-value penalty has a corner at zero, and optimization slides coefficients right into that corner, so the model does feature selection for free.

Lasso was the exciting one, and people found its limits fast. It can select at most $n$ features when you have more features than samples, which is a problem in exactly the wide datasets where you most want selection. And with a group of correlated features it tends to keep one and discard the rest more or less arbitrarily, which makes the selected set unstable. Hui Zou and Trevor Hastie named and solved this in 2005, in "Regularization and variable selection via the elastic net" — add both penalties, get sparsity from the L1 part and grouped, stable selection from the L2 part. They coined the grouping effect as the property that correlated features get similar coefficients instead of one surviving and the others dying. Five years later Friedman, Hastie and Tibshirani gave it the fast coordinate-descent solver (the glmnet algorithm) that scikit-learn still uses, and that we build a small version of below.

The intuition

Start with the problem elastic net is built for. Here are two of the diabetes features, s1 and s2 — total serum cholesterol and LDL, two blood measurements that rise and fall together. Plotted against each other they're almost a line: their correlation is 0.897.

Now think about what a fit does with two features this close. They carry almost the same information, so the model can hand the credit to s1, or to s2, or split it any way in between, and the training error barely notices. Ordinary least squares picks one of those infinitely many splits and gives you enormous, opposite-signed coefficients that cancel — unstable and unreadable. Lasso picks a corner: it keeps one of them and zeroes the other, which is at least sparse but throws away a real feature and depends on noise to decide which one dies. Ridge keeps both and makes them small and similar, which is stable but never sparse.

Elastic net's move is to do both at the same time. The L2 part pulls correlated features toward each other so they share the coefficient like ridge does — that's the grouping effect — while the L1 part is still free to zero out the features that carry nothing. You get a sparse model where correlated features enter and leave as a group, not one at a time by coin flip. That's the picture to hold: the mix ratio decides how much "keep correlated features together" you buy at the cost of how much "make the model sparse."

The math

Everything is the linear model from the regression chapter — predictions are a weighted sum of features plus an intercept:

with $w \in \mathbb{R}^{d}$ the coefficient vector and $b$ the intercept. What changes is the thing we minimize. Ordinary least squares minimizes just the error; elastic net adds a penalty on the size of the coefficients:

Two knobs. The penalty strength $\lambda \ge 0$ sets how hard the whole penalty pushes; at $\lambda = 0$ it vanishes and you're back to least squares. The mix ratio $\alpha \in [0, 1]$ splits that push between the two norms — the L1 norm $\lVert w\rVert_1 = \sum_j |w_j|$ and the squared L2 norm $\lVert w\rVert_2^2 = \sum_j w_j^2$. The $\frac{1}{2n}$ on the error and the $\frac{1}{2}$ on the L2 term are conveniences that make the derivative below come out clean; they also match scikit-learn's objective exactly, so the two fits are comparable coefficient for coefficient.

The mix ratio is the whole story. Set $\alpha = 1$ and the L2 term drops out entirely:

which is the lasso. Set $\alpha = 0$ and the L1 term drops out:

which is ridge regression. Elastic net is the convex combination of the two, and those are its endpoints.

To fit it we can't just take a gradient step, because the L1 norm has no derivative at zero — that corner is the very thing that produces sparsity, and it's also what breaks plain gradient descent. The trick is coordinate descent: freeze every coefficient but one and minimize the loss along that single coordinate, which is a one-dimensional problem with a closed-form solution. Working the derivative for coefficient $j$, with standardized features so the curvature $\frac{1}{n}\sum_i x_{ij}^2$ equals 1, gives the update

where $\rho_j$ is the correlation between feature $j$ and the residual with $j$'s own contribution added back in, and $S$ is the soft-threshold operator:

Read that update as the two penalties acting in sequence. The soft-threshold in the numerator is the L1 part: it shrinks $\rho_j$ toward zero by $\lambda\alpha$ and clamps anything smaller than that to exactly zero — this is what creates sparsity. The denominator $1 + \lambda(1-\alpha)$ is the L2 part: a constant factor that shrinks every coefficient toward zero without ever reaching it — this is what stabilizes correlated groups. Cycle that update over all the coefficients until they stop moving, and you've fit elastic net.

What it's good at, what it isn't

The case for elastic net is that it gives you lasso's sparsity without lasso's fragility. When features are correlated — and in real data they usually are — lasso's selection is unstable: the group of features it keeps can change when you resample, because the L1 penalty has no reason to prefer one member of a correlated group over another. The L2 part fixes that by tying correlated coefficients together, so you get a sparse model whose selected set actually holds still. It also handles the wide case, more features than samples, where pure lasso caps out at $n$ selected features and elastic net doesn't. Two knobs instead of one is the price, and cross-validation over a grid of $\lambda$ and $\alpha$ is the standard way to pay it.

The case against it is that regularization is not a free lunch and elastic net does not always earn its keep. On this diabetes data, with only ten features and plenty of samples, the penalty barely moves predictive accuracy — you'll see the test R² land in the mid-thirties whether you regularize or not, because there's no overfitting here to fix. The payoff isn't a better score; it's a model whose coefficients you can trust and read. And the two knobs interact in ways that aren't always intuitive: the same $\lambda$ means very different amounts of sparsity at different $\alpha$, so you can't tune one and forget the other. Reach for it when you have many correlated features and you want a sparse, stable model. Don't reach for it expecting accuracy you didn't otherwise have.

The data

The diabetes dataset ships with scikit-learn, so there's nothing to download. It has 442 patients, ten baseline features — age, sex, body-mass index, blood pressure, and six blood-serum measurements labelled s1 through s6 — and a target that measures disease progression one year after baseline. The features come already standardized: each column is mean-zero and scaled, which is exactly what a penalized model needs, since a penalty on coefficient size only makes sense when the features are on the same scale.

The single strongest feature is body-mass index, and the relationship is the one you'd guess — higher BMI, faster progression, with a lot of scatter around it. Here it is against the target:

The interesting part isn't BMI, though — it's the serum group. The six s-features are different blood chemistry readings, several of them strongly correlated, and s1 and s2 in particular sit at that 0.897 you saw above. That correlated cluster is what makes this a good elastic-net dataset: it's precisely where lasso and elastic net part ways.

Build it, one function at a time

Pure NumPy, in the order you'd write it: the model and its score first, then the two pieces of the update, then the loop that runs them. The features are assumed standardized going in, which is what lets the coordinate update stay this simple.

The model and the score are unchanged from ordinary linear regression — regularization changes how we choose the coefficients, never how we use them. Prediction is the features times the weights plus the intercept:

def predict(X, w, b):
    """The linear model: yhat = Xw + b.

    Same one-liner as ordinary least squares. Elastic net changes how we CHOOSE
    w and b, never how we use them. X is (n, d), w is (d,), b is a scalar.
    """
    return X @ w + b

And R² is the fraction of variance explained, the number we'll report — one minus our squared error over the error of always guessing the mean:

def r2_score(X, y, w, b):
    """Coefficient of determination: fraction of variance explained.

    1 minus (our squared error / the error of always guessing the mean). 1.0 is
    perfect, 0.0 is no better than the mean, negative is worse than the mean.
    """
    resid = y - predict(X, w, b)
    ss_res = float(np.sum(resid ** 2))
    ss_tot = float(np.sum((y - y.mean()) ** 2))
    return 1.0 - ss_res / ss_tot

Now the piece that makes elastic net different from ridge: the soft-threshold operator. This is the L1 penalty in code. It shrinks its input toward zero by a margin and clamps anything inside that margin to exactly zero. Ridge's smooth penalty can only scale a coefficient down; this one can switch it off:

def soft_threshold(z, gamma):
    """The L1 operator: shrink z toward zero by gamma, and clamp at zero.

    S(z, gamma) = sign(z) * max(|z| - gamma, 0). This is the whole reason
    elastic net (and lasso) produce exact zeros: if the unpenalized pull on a
    coefficient is weaker than gamma, the coefficient is set to precisely 0, not
    to something small. Ridge's smooth L2 penalty can't do this — it scales
    coefficients down but never crosses zero.
    """
    return np.sign(z) * max(abs(z) - gamma, 0.0)

It's worth writing the full objective out, both because it's what we're minimizing and because it's the honest way to check a fit — the loss has to go down every sweep. It's the mean squared error (halved) plus the blended penalty, with $\alpha$ splitting the penalty between the L1 and L2 terms:

def elastic_net_objective(X, y, w, b, lam, alpha):
    """The full elastic-net loss: MSE half plus the blended penalty.

    (1 / 2n) * ||y - Xw - b||^2  +  lambda * [ alpha * ||w||_1
                                             + (1 - alpha) / 2 * ||w||^2 ]

    The 1/2n scaling on the error and the 1/2 on the L2 term are conveniences
    that make the coordinate-descent update fall out clean; they also match
    scikit-learn's objective so the two fits can be compared coefficient for
    coefficient. alpha = 1 zeroes the L2 term (pure lasso); alpha = 0 zeroes the
    L1 term (pure ridge).
    """
    n = X.shape[0]
    resid = predict(X, w, b) - y
    mse_half = float(np.sum(resid ** 2)) / (2.0 * n)
    l1 = float(np.sum(np.abs(w)))
    l2 = float(np.sum(w ** 2))
    penalty = lam * (alpha * l1 + (1.0 - alpha) / 2.0 * l2)
    return mse_half + penalty

And here's the fit. Coordinate descent cycles over the coefficients one at a time; for each one it computes $\rho_j$, the correlation between that feature and the residual with the feature's own contribution added back, then applies the update from the math section — soft-threshold for the L1 part, divide by the shrink factor for the L2 part. The intercept stays out of the penalty, so we center the target and read the intercept off as its mean. It stops when a full sweep barely moves anything:

def coordinate_descent(X, y, lam, alpha, n_iters=1000, tol=1e-9):
    """Fit elastic net by cycling over coefficients, one closed-form step each.

    Standard trick for a non-differentiable (L1) penalty: you can't take a
    single gradient step, but you CAN minimize the loss exactly along one
    coordinate at a time while the others are held fixed. For coefficient j that
    1-D problem has a closed form,

        rho_j = (1/n) * x_j . (y - yhat + x_j * w_j)     # correlation with the
                                                         # partial residual
        w_j   = soft_threshold(rho_j, lambda * alpha)    # L1: shrink and clamp
                / (1 + lambda * (1 - alpha))             # L2: shrink factor

    Read the update as two effects in sequence. The soft-threshold is the L1
    part: it can push w_j to exactly zero. The denominator is the L2 part: a
    constant shrink that pulls every coefficient toward zero without ever
    reaching it — this is what stabilizes the fit when features are correlated,
    so a group of correlated features gets kept together rather than one being
    picked arbitrarily. Because features are standardized, the curvature term
    (1/n) * sum(x_j^2) is 1, which is why it doesn't appear.

    y is centered here, so the intercept is just the target mean; we return it
    alongside the weights. We cycle until the largest coefficient change in a
    full sweep is a negligible fraction of the largest coefficient.
    """
    n, d = X.shape
    b = float(y.mean())
    yc = y - b                                 # center: intercept is out of the penalty
    w = np.zeros(d)
    denom = 1.0 + lam * (1.0 - alpha)          # the L2 shrink factor
    for _ in range(n_iters):
        max_change = 0.0
        max_coef = 0.0
        for j in range(d):
            resid = yc - X @ w                 # residual with the current w
            rho = (1.0 / n) * X[:, j] @ (resid + X[:, j] * w[j])
            w_j = soft_threshold(rho, lam * alpha) / denom
            max_change = max(max_change, abs(w_j - w[j]))
            max_coef = max(max_coef, abs(w_j))
            w[j] = w_j
        if max_coef > 0.0 and max_change / max_coef < tol:
            break
    return w, b

That's the whole algorithm. Two effects, one line of update, wrapped in a loop.

Watch it work

This is the point of the chapter. We fix the penalty strength at $\lambda = 1$ and sweep the mix ratio $\alpha$ from 0 to 1 — from pure ridge on the left end to pure lasso on the right — refitting the whole model from scratch at every step. Each frame is one real fit; the bars are that fit's ten coefficients, blue for positive and orange for negative, with the dashed line at zero. The caption names $\alpha$, says which regime you're in, and counts how many coefficients are still nonzero.

Press play and watch the shape change. At $\alpha = 0$, ridge, all ten bars are up — nothing is zero, everything is small and stable. As $\alpha$ climbs, the L1 part starts biting: bars shrink, and one by one the weak ones snap to exactly zero and disappear. By $\alpha = 1$, lasso, three of them are gone and you're left with seven. Reset and run it as many times as you like; it's deterministic, the same sweep every time.

The thing to watch for is s2, the LDL feature — the one correlated at 0.897 with s1. Follow its bar as $\alpha$ rises. Under ridge and through most of the elastic-net range it stays up, sharing the load with s1. At the far right, under pure lasso, it drops to zero: lasso has decided the correlated pair only needs one member and cut s2 loose. That single bar vanishing is the grouping effect, or rather its absence — it's exactly what elastic net is built to prevent, and it's why the middle of this sweep is where you want to live when your features are correlated.

The full implementation

The whole file, no library — the model, R², the soft-threshold, the objective, and coordinate descent. This is the code the animation above actually ran:

"""Elastic net regression, built from scratch.

Elastic net is linear regression with two penalties stapled on: an L1 term that
drives coefficients to exactly zero (sparsity, lasso's trick) and an L2 term
that keeps them small and stable when features move together (ridge's trick). A
mix ratio alpha slides between the two — alpha = 1 is pure lasso, alpha = 0 is
pure ridge, and everything in between is elastic net.

We fit it by coordinate descent: cycle over one coefficient at a time and solve
its 1-D subproblem in closed form. That subproblem is a soft-threshold (the L1
part, which can snap a coefficient to zero) divided by a shrink factor (the L2
part, which never quite does). Pure NumPy — no ML library in this file. The
`# region:` markers are what the chapter's include directives pull in.

Inputs are assumed standardized: each feature column mean-zero and unit
variance, so the per-coordinate curvature (1/n) * sum(x_j^2) is 1 and every
coefficient is penalized on the same footing. Standardizing is not optional for
a penalized model — without it lambda punishes big-scale features less than
small-scale ones purely by accident.
"""

import numpy as np
import pandas as pd


# region: predict
def predict(X, w, b):
    """The linear model: yhat = Xw + b.

    Same one-liner as ordinary least squares. Elastic net changes how we CHOOSE
    w and b, never how we use them. X is (n, d), w is (d,), b is a scalar.
    """
    return X @ w + b
# endregion


# region: r2
def r2_score(X, y, w, b):
    """Coefficient of determination: fraction of variance explained.

    1 minus (our squared error / the error of always guessing the mean). 1.0 is
    perfect, 0.0 is no better than the mean, negative is worse than the mean.
    """
    resid = y - predict(X, w, b)
    ss_res = float(np.sum(resid ** 2))
    ss_tot = float(np.sum((y - y.mean()) ** 2))
    return 1.0 - ss_res / ss_tot
# endregion


# region: soft_threshold
def soft_threshold(z, gamma):
    """The L1 operator: shrink z toward zero by gamma, and clamp at zero.

    S(z, gamma) = sign(z) * max(|z| - gamma, 0). This is the whole reason
    elastic net (and lasso) produce exact zeros: if the unpenalized pull on a
    coefficient is weaker than gamma, the coefficient is set to precisely 0, not
    to something small. Ridge's smooth L2 penalty can't do this — it scales
    coefficients down but never crosses zero.
    """
    return np.sign(z) * max(abs(z) - gamma, 0.0)
# endregion


# region: objective
def elastic_net_objective(X, y, w, b, lam, alpha):
    """The full elastic-net loss: MSE half plus the blended penalty.

    (1 / 2n) * ||y - Xw - b||^2  +  lambda * [ alpha * ||w||_1
                                             + (1 - alpha) / 2 * ||w||^2 ]

    The 1/2n scaling on the error and the 1/2 on the L2 term are conveniences
    that make the coordinate-descent update fall out clean; they also match
    scikit-learn's objective so the two fits can be compared coefficient for
    coefficient. alpha = 1 zeroes the L2 term (pure lasso); alpha = 0 zeroes the
    L1 term (pure ridge).
    """
    n = X.shape[0]
    resid = predict(X, w, b) - y
    mse_half = float(np.sum(resid ** 2)) / (2.0 * n)
    l1 = float(np.sum(np.abs(w)))
    l2 = float(np.sum(w ** 2))
    penalty = lam * (alpha * l1 + (1.0 - alpha) / 2.0 * l2)
    return mse_half + penalty
# endregion


# region: coordinate_descent
def coordinate_descent(X, y, lam, alpha, n_iters=1000, tol=1e-9):
    """Fit elastic net by cycling over coefficients, one closed-form step each.

    Standard trick for a non-differentiable (L1) penalty: you can't take a
    single gradient step, but you CAN minimize the loss exactly along one
    coordinate at a time while the others are held fixed. For coefficient j that
    1-D problem has a closed form,

        rho_j = (1/n) * x_j . (y - yhat + x_j * w_j)     # correlation with the
                                                         # partial residual
        w_j   = soft_threshold(rho_j, lambda * alpha)    # L1: shrink and clamp
                / (1 + lambda * (1 - alpha))             # L2: shrink factor

    Read the update as two effects in sequence. The soft-threshold is the L1
    part: it can push w_j to exactly zero. The denominator is the L2 part: a
    constant shrink that pulls every coefficient toward zero without ever
    reaching it — this is what stabilizes the fit when features are correlated,
    so a group of correlated features gets kept together rather than one being
    picked arbitrarily. Because features are standardized, the curvature term
    (1/n) * sum(x_j^2) is 1, which is why it doesn't appear.

    y is centered here, so the intercept is just the target mean; we return it
    alongside the weights. We cycle until the largest coefficient change in a
    full sweep is a negligible fraction of the largest coefficient.
    """
    n, d = X.shape
    b = float(y.mean())
    yc = y - b                                 # center: intercept is out of the penalty
    w = np.zeros(d)
    denom = 1.0 + lam * (1.0 - alpha)          # the L2 shrink factor
    for _ in range(n_iters):
        max_change = 0.0
        max_coef = 0.0
        for j in range(d):
            resid = yc - X @ w                 # residual with the current w
            rho = (1.0 / n) * X[:, j] @ (resid + X[:, j] * w[j])
            w_j = soft_threshold(rho, lam * alpha) / denom
            max_change = max(max_change, abs(w_j - w[j]))
            max_coef = max(max_coef, abs(w_j))
            w[j] = w_j
        if max_coef > 0.0 and max_change / max_coef < tol:
            break
    return w, b
# endregion


def standardize(X_train, X_other=None):
    """Z-score using the training mean/std so test data can't peek.

    Returns the standardized train matrix (and, if given, the other matrix
    scaled by the SAME statistics). Population std matches the (1/n) curvature
    the coordinate-descent update assumes.
    """
    mu = X_train.mean(axis=0)
    sd = X_train.std(axis=0)
    Xtr = (X_train - mu) / sd
    if X_other is None:
        return Xtr, mu, sd
    return Xtr, (X_other - mu) / sd, mu, sd


def load_data(path="../data/diabetes.csv"):
    """Diabetes: 442 patients, 10 standardized features, one-year progression.

    Returns (X, y, feature_names). y is a quantitative measure of disease
    progression a year after baseline. The serum-cholesterol features (s1..s6)
    are strongly correlated — s1 and s2 sit at 0.90 — which is exactly the mess
    elastic net is built for.
    """
    df = pd.read_csv(path)
    target = "target"
    features = [c for c in df.columns if c != target]
    X = df[features].to_numpy(float)
    y = df[target].to_numpy(float)
    return X, y, features

The library version

Nobody hand-rolls coordinate descent in production. sklearn.linear_model.ElasticNet is the same model with the same objective, fit by the same algorithm — the glmnet coordinate descent — just written in Cython and hardened. The one thing to get right is the naming, because scikit-learn's parameters collide with the math. What we call $\lambda$, the penalty strength, sklearn calls alpha. What we call $\alpha$, the L1/L2 mix, sklearn calls l1_ratio. So our $\lambda = 1$, $\alpha = 0.5$ is sklearn's alpha=1.0, l1_ratio=0.5:

def sklearn_fit(X_train, y_train, lam, alpha):
    """Fit elastic net with sklearn. `lam` is our penalty strength (sklearn's
    `alpha`); `alpha` is our L1/L2 mix (sklearn's `l1_ratio`). Returns
    (weights, intercept) so the coefficients line up one for one with ours."""
    model = ElasticNet(
        alpha=lam, l1_ratio=alpha,
        fit_intercept=True, max_iter=200000, tol=1e-12,
    )
    model.fit(X_train, y_train)
    return model.coef_, float(model.intercept_)

Scoring is fit on train, report R² on the held-out test set — the face-off number:

def sklearn_score(X_train, y_train, X_test, y_test, lam, alpha):
    """Fit on train, report test R^2 — the face-off number."""
    model = ElasticNet(
        alpha=lam, l1_ratio=alpha,
        fit_intercept=True, max_iter=200000, tol=1e-12,
    )
    model.fit(X_train, y_train)
    return float(model.score(X_test, y_test))

Because it's the same objective solved by the same algorithm on the same standardized data, it lands on the same coefficients we do. The only differences that matter in practice are the ones we're not exercising here: sklearn's l1_ratio has to be strictly positive (the pure-ridge endpoint uses Ridge instead), and it defaults to standardizing nothing, so you're responsible for scaling your features before you hand them over — which we've done.

Scratch versus library

Split the diabetes data 80/20 — 353 patients to train on, 89 held out — fit elastic net at $\lambda = 1$, $\alpha = 0.5$ on the training half with both implementations, and score on the held-out half:

Both bars are the same length: test R² of 0.3653. Not close — identical. Our coordinate descent matches sklearn's coefficients to fourteen decimal places, because there's one minimizer of a convex objective and every correct solver finds it. The exact figures live in results.json, regenerated whenever the code changes, so the prose and the chart can't drift from what the code produced. On this split the fit keeps nine of the ten features — it's zeroed one — and both implementations zero the same one.

To see what that R² looks like as predictions, plot the fit against the truth on the held-out patients — predicted progression on the vertical, actual on the horizontal, with the dashed diagonal marking a perfect fit. Points on the line are exactly right; the vertical spread off it is the error the R² is measuring:

The cloud tracks the diagonal — the fit has the trend right — but it's a wide cloud, and it's flatter than the line: low actuals get over-predicted and high actuals under-predicted, the regression-toward-the-mean you'd expect from a model that explains a third of the variance. An R² of 0.37 is modest, and that's the honest headline: on data this small and this un-overfit, the penalty doesn't buy you accuracy. What it buys is the coefficient structure, and that's the chart worth looking at. Here are the ten coefficients under all three regimes — ridge, elastic net, and lasso — fit on the full standardized data at the same $\lambda = 1$:

Look at s1 and s2, the correlated pair. Ridge keeps both, at 0.28 and -1.40, sharing the signal between them. Elastic net keeps both too, at -0.24 and -2.37 — the L2 part holding the group together. Lasso pours everything into s1 at -4.84 and cuts s2 to exactly zero: handed a correlated pair, it took one and dropped the other. That's the grouping effect in one chart. Ridge keeps all ten features but gives you no sparsity; lasso gives you sparsity but breaks the correlated group; elastic net sits between them, sparse where it can be and grouped where it should be.

Takeaways

Reach for elastic net when you have many correlated features and you want a sparse model you can trust. That's the specific situation it wins, and it's a common one — real feature sets are full of near-duplicates, and lasso's habit of keeping one of them at random and zeroing the rest makes its selected set unstable in a way that quietly undermines whatever you build on top of it. Elastic net's L2 part ties those correlated features together so the selection holds still across resamples, while the L1 part still hands you the zeros. It's the safe default among the penalized linear models for exactly the data that breaks lasso.

Be honest about what it does and doesn't buy. On this diabetes fit the test R² was 0.37 with the penalty and would be about the same without it — regularization didn't rescue a model that was drowning in overfitting, because there was no overfitting to rescue. What it delivered was structure: a coefficient vector where the correlated cholesterol features rise and fall together instead of one arbitrarily surviving. When your model has many more features than the samples can pin down, that same mechanism starts paying off in accuracy too, but the reason to use it is the stability, and the accuracy is a bonus when it comes.

And keep the two endpoints in your head, because they're the two chapters on either side of this one. Slide the mix ratio to zero and you have ridge, all shrink and no selection; slide it to one and you have lasso, all selection and no grouping. Elastic net isn't a third thing — it's the dial between them, and the right setting is a cross-validation search, not a guess. Start it near the middle when your features are correlated, push it toward lasso when you want a leaner model and can afford the instability, and toward ridge when you can't afford any. The animation in the middle of this chapter is that dial; the whole skill is knowing where on it your data wants to sit.