Support vector machines with SGD

What this chapter covers

Logistic regression drew a straight boundary and asked every point for a probability. The support vector machine draws a straight boundary too, but it asks a different question: not "how likely is each label" but "where's the widest empty corridor I can run between the two classes." It only cares about the points near the edge — the ones that would get bumped if the line moved — and it ignores everything sitting safely in the interior. That single idea, maximize the margin, turns out to be one of the most durable in machine learning.

We build the linear SVM from scratch in NumPy: the decision function, the hinge loss that penalizes points for crowding the margin, the L2 term that controls how wide the margin gets, the subgradient, and a stochastic gradient descent loop that walks the whole thing downhill one minibatch at a time. Then we let scikit-learn's SGDClassifier(loss="hinge") fit the same model and check the numbers line up. The centerpiece, like last chapter, is watching it train — but this time you're watching a corridor. Each frame draws the boundary and its two margin lines over a 2-D dataset, rings the support vectors, marks the mistakes, and traces the hinge loss dropping underneath. The margin sweeps in from the edges and settles into the gap between the classes, and by the end you can see exactly which handful of points are holding the line in place.

A bit of history

The margin idea is old and it comes from theory, not from a benchmark. Vladimir Vapnik and Alexey Chervonenkis worked it out in the Soviet Union in 1963, as part of the statistical learning theory they were building — the VC theory that later carried both their initials. Their result was that a classifier's ability to generalize is tied to the margin it leaves around the boundary: a wide, confident gap between the classes is not just prettier, it's provably less likely to overfit. That's a stronger claim than "it works on my data," and it's why the SVM arrived with a reputation.

The method most people know took two more steps, both with Vapnik in the room. In 1992 Bernhard Boser, Isabelle Guyon, and Vapnik added the kernel trick — a way to get a curved boundary by measuring similarity in a transformed space without ever building that space explicitly. Then in 1995 Corinna Cortes and Vapnik published the soft-margin SVM, which lets a few points sit inside the margin or on the wrong side in exchange for a boundary that isn't wrecked by a single outlier. That soft margin, trained on a hinge loss with an L2 penalty, is exactly the model we build here. Through the late 1990s and 2000s the SVM was the default strong classifier, the thing you had to beat. The one wrinkle was training cost: the textbook SVM is a quadratic program that scales badly with the number of examples. The fix that made it scale is the one in this chapter's title — in 2007 Shai Shalev-Shwartz and coauthors published Pegasos, which showed you could train an SVM by plain stochastic gradient descent on the hinge loss and reach the same solution far faster. We do the linear, SGD form here. The kernel trick is a natural extension — I'll point at it at the end, but we won't implement it.

The intuition

Here's a 2-D dataset, two classes, some overlap. Two features on the axes, color for the true label. These are the points the SVM will run its corridor between.

Plenty of straight lines separate most of these points. Logistic regression would have picked one by asking every point to vote on a probability. The SVM picks differently. Imagine widening a band around your candidate line — pushing two parallel walls apart, one on each side, until they bump into the nearest points. The line the SVM wants is the one whose band is widest. Everything in the middle of a blob has no say; it could move a long way without touching a wall. Only the points right at the edges — the ones the walls rest against — matter. Those are the support vectors, and they're the whole model. Delete every other point and you'd get the identical boundary.

That's the shift in mindset worth carrying. Logistic regression is a model of all the data. An SVM is a model of the boundary. When the classes are cleanly separable, the second framing gives you a more stable line, because it's anchored to the few points that actually define the frontier instead of being tugged on by every point in the cloud.

The math

Start with the same linear score as last chapter. For a feature vector $x_i$ with $D$ entries, weights $w$, and bias $b$:

The boundary is the flat surface $f(x) = 0$. We predict a label by its sign, and here the labels live in $\{-1, +1\}$ rather than $\{0, 1\}$ — that choice is what makes everything below come out clean:

Now the margin. The functional margin of a point is $y_i\, f(x_i)$: positive when the point is on the correct side, negative when it's misclassified, and larger the more confident the call. The geometric margin — the actual distance from the point to the boundary — is that quantity divided by $\lVert w \rVert$. We fix the scale by declaring the margin lines to be the two surfaces $f(x) = +1$ and $f(x) = -1$. The perpendicular distance between them works out to

so maximizing the margin means minimizing $\lVert w \rVert$. That's the whole trick hiding in plain sight: a smaller weight vector is a wider corridor.

If the classes were perfectly separable we'd just minimize $\lVert w \rVert$ subject to every point sitting at or beyond its margin, $y_i f(x_i) \ge 1$. Real data overlaps, so we relax that hard constraint into a penalty — the hinge loss:

Read it off the margin. A point with $y_i f(x_i) \ge 1$ sits at or past its margin line and pays nothing. A point inside the margin ($0 < y_i f < 1$) or misclassified ($y_i f < 0$) pays linearly in how far it fell short. No reward for being extra far past the margin — once you're clear, you're clear. Putting the two halves together gives the soft-margin objective SGD minimizes:

The regularization strength $\lambda$ is the one dial that matters: crank it up and the model prizes a wide margin, tolerating more points inside it; turn it down and the margin narrows to avoid violations. It's the same bias-variance knob under a different name.

The hinge has a corner at $y_i f = 1$, so it isn't differentiable everywhere and we use a subgradient. Only the points that are still paying — the ones with $y_i f(x_i) < 1$ — push the boundary:

The regularization term $\lambda w$ always pulls the weights toward zero, widening the margin; the data term shoves the boundary away from any point crowding it. SGD estimates that sum on a small random minibatch instead of the whole set, which is what lets it scale. Note the bias $b$ gets no $\lambda$ term — we regularize the orientation of the plane, not where it sits.

What it's good at, what it isn't

The SVM's strength is the margin, and it's a real one. A boundary chosen to sit as far as possible from both classes tends to generalize well — that's not folklore, it's what the VC theory predicts and what you feel in practice as robustness. Because the model depends only on the support vectors, it's stable: points deep inside a class can shift or vanish and the boundary doesn't flinch. And with the kernel trick bolted on, the same machinery draws highly nonlinear boundaries, which is what made SVMs the strong general-purpose classifier for a decade. The hinge loss is a good citizen too — convex, so training has one basin, and it stops caring about a point the moment it's safely classified, unlike a loss that keeps chasing already-correct examples.

The costs are the flip side. The plain SVM hands you a label, not a probability — there's no calibrated $P(y=1)$ falling out of it the way it does from logistic regression, and the usual fix (Platt scaling) is a bolt-on you have to trust. The kernel version scales poorly: the quadratic program is roughly quadratic to cubic in the number of examples, which is precisely why SGD training exists and why the kernel trick quietly fell out of fashion once datasets got large. And it has a real hyperparameter to tune in $\lambda$ (or $C$, its inverse), plus the kernel choice if you go nonlinear — more knobs than a logistic regression you can fit and forget. The feature scaling matters more here than almost anywhere, which is the next thing.

The data

For this chapter the dataset is deliberately small and two-dimensional so the geometry is legible: 120 points, two classes of 60, generated as a pair of Gaussian blobs with enough spread that they overlap in the middle. A straight line separates most of them and a handful sit on the wrong side — which is what we want, so there are honest support vectors and a couple of genuine mistakes to watch. The raw data lives in data/blobs.csv; the README notes exactly how it was generated and seeded.

One preprocessing step is not optional for an SVM: standardize the features first. The model measures distance to a hyperplane, and distance is dominated by whichever feature happens to have the largest numeric range. Leave one feature in the hundreds and another in fractions, and the margin quietly contorts itself to please the big one — the geometry stops meaning what you think it means. So we subtract each column's mean and divide by its standard deviation, computed on the training data and reapplied to any held-out data, so every feature contributes to the distance on equal terms. Our blobs already sit on similar scales, so standardizing barely moves them here, but the habit is one to build now, because on real tabular data skipping it will silently ruin an SVM in a way it merely inconveniences a tree.

Build it, one function at a time

Seven small pieces, bottom up. Start with the score, the same linear combination logistic regression used before it squashed anything.

def decision_function(X, w, b):
    """The raw signed score f(x) = w·x + b, one per row.

    X is (N, D), w is (D,), b is a scalar. This is the distance-to-boundary
    signal before we take its sign: positive on one side of the hyperplane,
    negative on the other, zero exactly on it. Its magnitude is how far the
    point sits from the boundary, in units of 1 / ||w||.
    """
    return X @ w + b

To turn scores into labels, take the sign. This is where the $\{-1, +1\}$ convention earns its keep — the label is just which side of zero the score lands on, no threshold to name.

def predict(X, w, b):
    """Label each row by which side of the boundary it falls on.

    Return +1 when the score is non-negative, -1 otherwise — the sign of the
    decision function. Ties (a point exactly on the boundary) go to +1.
    """
    return np.where(decision_function(X, w, b) >= 0, 1, -1)

Now the loss. The hinge is the heart of the model: it reads each point's margin and charges it only if it's crowding the corridor.

def hinge_loss(X, y, w, b):
    """Mean hinge loss over the dataset, labels in {-1, +1}.

    Per point: max(0, 1 - y·f(x)). The quantity y·f(x) is the *margin* of the
    point — positive when it's classified correctly, and >= 1 when it sits at
    or beyond the margin line, where it costs nothing. A point inside the
    margin (0 < y·f < 1) or misclassified (y·f < 0) pays linearly. That
    hinge is what pushes the boundary until the classes are cleanly separated
    with room to spare.
    """
    margins = y * decision_function(X, w, b)
    return float(np.mean(np.maximum(0.0, 1.0 - margins)))

The hinge alone would happily let $\lVert w \rVert$ run off to infinity, since a bigger weight vector makes every correct point look more confident. The L2 term reins that in, and the two together are the full objective — the thing training actually minimizes:

def objective(X, y, w, b, lam):
    """The full soft-margin objective SGD minimizes.

    Regularized hinge loss: a data term that wants every point past its margin,
    plus an L2 term (lam/2)·||w||² that wants w small. Shrinking ||w|| widens
    the margin (its width is 2 / ||w||), so lam is the dial that trades margin
    width against training mistakes. Small lam → narrow margin, few violations;
    large lam → wide margin, more points allowed inside it. The bias b is not
    regularized — only the orientation of the plane pays the penalty.
    """
    reg = 0.5 * lam * float(w @ w)
    return reg + hinge_loss(X, y, w, b)

That lam is the whole personality of the model. Big lam keeps w small and the margin wide; small lam lets w grow to eliminate violations. Next the gradient — a subgradient, because the hinge has that corner. Only the points still paying into the loss contribute:

def subgradient(Xb, yb, w, b, lam):
    """(Sub)gradient of the objective over a minibatch.

    The hinge has a kink at y·f = 1, so we use a subgradient: only the points
    with margin < 1 (inside the margin or misclassified) push the boundary;
    the rest contribute nothing to the data term. For those active points the
    gradient of the hinge is -y·x for w and -y for b. The L2 term always adds
    lam·w. Averaging the data term over the minibatch is what makes this
    *stochastic* — a noisy but cheap estimate of the full gradient.
    """
    margins = yb * (Xb @ w + b)
    active = margins < 1.0                       # points still paying hinge
    n = len(yb)
    dw = lam * w - (Xb[active].T @ yb[active]) / n
    db = -float(np.sum(yb[active])) / n          # bias is not regularized
    return dw, db

Look at what's there and what isn't. The active mask picks out exactly the points inside or on the wrong side of the margin — the support vectors in the making — and only they push the boundary. Everything comfortably classified drops out of the sum entirely. That masking is the SVM's defining move written as one line of NumPy.

Now the training loop. Start the plane flat, shuffle each epoch with a seeded generator so the run is reproducible, and take one subgradient step per minibatch:

def fit(X, y, lam=0.01, lr=0.1, n_epochs=45, batch_size=12, seed=0, record=False):
    """Train the SVM by stochastic gradient descent.

    Start the plane flat (w = 0, b = 0). Each epoch, shuffle the rows with a
    seeded generator (so runs are reproducible), walk them in minibatches, and
    take one subgradient step per batch. With `record=True` we snapshot
    (w, b, hinge, objective) at the end of every epoch so the chapter can
    replay the margin widening frame by frame.
    """
    rng = np.random.default_rng(seed)
    N, D = X.shape
    w = np.zeros(D)
    b = 0.0
    history = []

    def snap():
        history.append((w.copy(), b,
                        hinge_loss(X, y, w, b),
                        objective(X, y, w, b, lam)))

    if record:
        snap()
    for _ in range(n_epochs):
        idx = rng.permutation(N)
        for start in range(0, N, batch_size):
            bi = idx[start:start + batch_size]
            dw, db = subgradient(X[bi], y[bi], w, b, lam)
            w -= lr * dw
            b -= lr * db
        if record:
            snap()
    if record:
        return w, b, history
    return w, b

This is what puts the "SGD" in the chapter title. Batch gradient descent would sum the gradient over all 120 points before every step; stochastic descent estimates it from twelve at a time and steps far more often. On this toy set the difference is academic, but it's the entire reason SVMs scale to millions of examples — you never have to touch the whole dataset to take a step. The record flag stashes the state at the end of each epoch so we can replay the training.

Last piece, the one that makes this an SVM and not just a linear classifier: which points are the support vectors. Anything with margin $\le 1$ — on the line, inside it, or misclassified.

def support_vectors(X, y, w, b, tol=1e-6):
    """Boolean mask of the support vectors: points with margin y·f(x) <= 1.

    These are the only points that shape the boundary — the ones on the margin
    line, inside it, or misclassified. Move any of them and the plane moves;
    delete a point comfortably beyond its margin and nothing changes. That's
    the whole idea of a support vector machine: the decision surface is
    supported by a handful of points, not the whole dataset.
    """
    margins = y * decision_function(X, w, b)
    return margins <= 1.0 + tol

Watch it work

This is the training itself, on the 2-D set so every point is visible. The top panel is the data; the solid orange line is the current decision boundary, and the two dashed lines are the margins, $f(x) = \pm 1$. A point gets an amber ring the moment it becomes a support vector — on or inside a margin line — and a red cross if the boundary actually misclassifies it. The bottom panel traces the hinge loss, one dot per epoch.

Press play. Every frame is one real epoch out of fit, the same weights the code produced. Reset and run it again as many times as you like.

Watch the first move. The plane starts flat — $w = 0$, no line at all — so there's no corridor yet and, for one frame, every one of the 120 points counts as a support vector because nothing has a margin. Then the gradient kicks in, a boundary appears, and the two margin lines come with it. Here's the thing to notice, and it's the opposite of the usual "the margin widens" story: because we started from a flat plane, the margin starts enormous — $\lVert w \rVert$ is tiny, and margin width is $2 / \lVert w \rVert$ — and it sweeps inward, the two dashed lines closing from the edges of the plot onto the gap between the classes. Same equilibrium the textbook draws, approached from the wide side instead of the narrow one. As it settles you can watch the support-vector count collapse: from all 120, down through the dozens, until only the points genuinely near the frontier keep their rings. By the last epoch the hinge loss has flattened at 0.156, the margin has settled to a width of 1.27, 33 points remain support vectors, and 6 stubborn points inside the overlap are still misclassified — training accuracy 0.95. The loss curve tells the same story from the side: a cliff in the first few epochs while the big violations get fixed, then a long flat glide as the boundary fine-tunes against the support vectors.

The six red crosses at the end are worth sitting with. The SVM didn't fail to place them — it chose to eat those six mistakes because the alternative, contorting the boundary to catch them, would have cost more margin than it was worth. That trade, a few violations for a wide clean corridor, is the soft margin doing exactly its job, and $\lambda$ is the knob that sets the exchange rate.

The full implementation

The whole file, top to bottom — the decision function, the hinge, the objective, the subgradient, the SGD loop, and the support-vector, accuracy, and standardize helpers the traces use. This is the code the animation above actually ran:

"""Linear support vector machine, built from scratch.

The soft-margin SVM: find the hyperplane that separates the two classes with
the widest possible margin, paying a hinge penalty for every point that lands
inside the margin or on the wrong side. We train it by stochastic gradient
descent on the regularized hinge loss — no quadratic program, no library, just
the (sub)gradient and a step size.

Labels live in {-1, +1} (not {0, 1}) — that convention is what makes the hinge
loss and the margin come out clean. Pure NumPy (+ pandas for loading). Every
function below appears in the chapter one step at a time; the `# region:`
markers are what the book's include directives pull in.
"""

import numpy as np
import pandas as pd


# region: decision_function
def decision_function(X, w, b):
    """The raw signed score f(x) = w·x + b, one per row.

    X is (N, D), w is (D,), b is a scalar. This is the distance-to-boundary
    signal before we take its sign: positive on one side of the hyperplane,
    negative on the other, zero exactly on it. Its magnitude is how far the
    point sits from the boundary, in units of 1 / ||w||.
    """
    return X @ w + b
# endregion


# region: predict
def predict(X, w, b):
    """Label each row by which side of the boundary it falls on.

    Return +1 when the score is non-negative, -1 otherwise — the sign of the
    decision function. Ties (a point exactly on the boundary) go to +1.
    """
    return np.where(decision_function(X, w, b) >= 0, 1, -1)
# endregion


# region: hinge_loss
def hinge_loss(X, y, w, b):
    """Mean hinge loss over the dataset, labels in {-1, +1}.

    Per point: max(0, 1 - y·f(x)). The quantity y·f(x) is the *margin* of the
    point — positive when it's classified correctly, and >= 1 when it sits at
    or beyond the margin line, where it costs nothing. A point inside the
    margin (0 < y·f < 1) or misclassified (y·f < 0) pays linearly. That
    hinge is what pushes the boundary until the classes are cleanly separated
    with room to spare.
    """
    margins = y * decision_function(X, w, b)
    return float(np.mean(np.maximum(0.0, 1.0 - margins)))
# endregion


# region: objective
def objective(X, y, w, b, lam):
    """The full soft-margin objective SGD minimizes.

    Regularized hinge loss: a data term that wants every point past its margin,
    plus an L2 term (lam/2)·||w||² that wants w small. Shrinking ||w|| widens
    the margin (its width is 2 / ||w||), so lam is the dial that trades margin
    width against training mistakes. Small lam → narrow margin, few violations;
    large lam → wide margin, more points allowed inside it. The bias b is not
    regularized — only the orientation of the plane pays the penalty.
    """
    reg = 0.5 * lam * float(w @ w)
    return reg + hinge_loss(X, y, w, b)
# endregion


# region: subgradient
def subgradient(Xb, yb, w, b, lam):
    """(Sub)gradient of the objective over a minibatch.

    The hinge has a kink at y·f = 1, so we use a subgradient: only the points
    with margin < 1 (inside the margin or misclassified) push the boundary;
    the rest contribute nothing to the data term. For those active points the
    gradient of the hinge is -y·x for w and -y for b. The L2 term always adds
    lam·w. Averaging the data term over the minibatch is what makes this
    *stochastic* — a noisy but cheap estimate of the full gradient.
    """
    margins = yb * (Xb @ w + b)
    active = margins < 1.0                       # points still paying hinge
    n = len(yb)
    dw = lam * w - (Xb[active].T @ yb[active]) / n
    db = -float(np.sum(yb[active])) / n          # bias is not regularized
    return dw, db
# endregion


# region: fit
def fit(X, y, lam=0.01, lr=0.1, n_epochs=45, batch_size=12, seed=0, record=False):
    """Train the SVM by stochastic gradient descent.

    Start the plane flat (w = 0, b = 0). Each epoch, shuffle the rows with a
    seeded generator (so runs are reproducible), walk them in minibatches, and
    take one subgradient step per batch. With `record=True` we snapshot
    (w, b, hinge, objective) at the end of every epoch so the chapter can
    replay the margin widening frame by frame.
    """
    rng = np.random.default_rng(seed)
    N, D = X.shape
    w = np.zeros(D)
    b = 0.0
    history = []

    def snap():
        history.append((w.copy(), b,
                        hinge_loss(X, y, w, b),
                        objective(X, y, w, b, lam)))

    if record:
        snap()
    for _ in range(n_epochs):
        idx = rng.permutation(N)
        for start in range(0, N, batch_size):
            bi = idx[start:start + batch_size]
            dw, db = subgradient(X[bi], y[bi], w, b, lam)
            w -= lr * dw
            b -= lr * db
        if record:
            snap()
    if record:
        return w, b, history
    return w, b
# endregion


# region: support_vectors
def support_vectors(X, y, w, b, tol=1e-6):
    """Boolean mask of the support vectors: points with margin y·f(x) <= 1.

    These are the only points that shape the boundary — the ones on the margin
    line, inside it, or misclassified. Move any of them and the plane moves;
    delete a point comfortably beyond its margin and nothing changes. That's
    the whole idea of a support vector machine: the decision surface is
    supported by a handful of points, not the whole dataset.
    """
    margins = y * decision_function(X, w, b)
    return margins <= 1.0 + tol
# endregion


def accuracy(X, y, w, b):
    """Fraction of rows the model labels correctly."""
    return float((predict(X, w, b) == y).mean())


def standardize(X, mean=None, std=None):
    """Center and scale each column to mean 0, unit variance.

    SVMs measure distance to a hyperplane, so a feature on a larger numeric
    scale silently dominates that distance and the margin tilts to please it.
    Standardizing puts every feature on equal footing before training; the same
    mean and std must then be applied to any held-out data. Returns the
    transformed X plus the mean and std so the test set gets the identical shift.
    """
    if mean is None:
        mean = X.mean(axis=0)
    if std is None:
        std = X.std(axis=0)
        std = np.where(std == 0, 1.0, std)
    return (X - mean) / std, mean, std


def load_data(path="../data/blobs.csv"):
    """Load the committed 2-D two-class dataset. Labels are in {-1, +1}."""
    df = pd.read_csv(path)
    X = df[["x1", "x2"]].to_numpy(float)
    y = df["label"].to_numpy(int)
    return X, y

The library version

You wouldn't hand-roll this for production, and you don't need to. Scikit-learn's SGDClassifier(loss="hinge") is the same model — a linear soft-margin SVM trained by stochastic gradient descent on exactly the objective above — and it hands back the fitted weights and bias the same way:

def sklearn_svm(Xtr, ytr, Xte, yte, alpha=0.01):
    """Fit sklearn's SGDClassifier(loss="hinge") on standardized features.

    `alpha` is the L2 strength — the same role as our `lam`. We hand it the
    already-standardized arrays so it's judged on exactly the footing as our
    from-scratch model. Returns the learned weights, bias, and test accuracy.
    """
    clf = SGDClassifier(loss="hinge", penalty="l2", alpha=alpha,
                        max_iter=2000, tol=1e-4, random_state=0)
    clf.fit(Xtr, ytr)
    w = clf.coef_[0]
    b = float(clf.intercept_[0])
    acc = float(clf.score(Xte, yte))
    return w, b, acc

A few things differ under the hood. Sklearn's alpha is our lam, the L2 strength, and we set them equal so the comparison is honest. Its learning rate isn't fixed the way ours is — by default it uses an adaptive schedule that shrinks the step over time, which is closer to the Pegasos recipe and usually converges in fewer passes. It also does its own shuffling and has an early-stopping tolerance. We feed it the already-standardized arrays so it's judged on identical footing — same features, same scaling, same split.

There's a second library worth naming, because it's the SVM most people learned first. LinearSVC solves the same soft-margin problem, but as the quadratic program Cortes and Vapnik originally posed — an exact solver rather than an SGD approximation. It's the right tool when the dataset is small enough that you'd rather have the exact margin than trade accuracy for speed, and it makes a good reference point: if our SGD is doing its job, it should land right next to LinearSVC's answer.

def sklearn_linsvc(Xtr, ytr, Xte, yte, C=1.0):
    """Fit LinearSVC — the same SVM solved as a quadratic program, not by SGD.

    A reference point: if our SGD is doing its job it should land near this
    exact-solver accuracy. `C` is the inverse of regularization strength.
    """
    clf = LinearSVC(C=C, max_iter=5000, random_state=0)
    clf.fit(Xtr, ytr)
    return float(clf.score(Xte, yte))

Scratch versus library

Split the data 70/30 — 84 points to train on, 36 held out — standardize on the training statistics, fit all three models, and score on the held-out set. Here's the face-off:

All three land on 0.9444 — our from-scratch SGD, sklearn's SGDClassifier, and the exact LinearSVC solver agree to four decimals on the held-out set. That's the result you want. It's not that they get lucky on the same 34 of 36 points by coincidence; it's that this is a convex problem with essentially one good boundary, and three different roads all arrive at it. The weights back that up: our SVM lands on $w = [1.31, 0.97]$ with bias $0.06$, and SGDClassifier on $w = [1.30, 0.97]$ with bias $0.04$ — the same plane to two decimals, from two different optimizers. When your hand-written subgradient lands on the library's answer, you've understood the model rather than just imported it.

On the training split our SVM leans on 25 support vectors out of 84 — call it a third of the data doing all the work while the other two-thirds sit far enough from the frontier to be irrelevant. That's the compression the SVM buys you: a boundary defined by a minority of the points. The honest caveats are the usual ones. This is a small, low-dimensional, roughly balanced dataset, so accuracy is a fair summary here in a way it wouldn't be on a skewed problem; and the held-out split is what keeps the number meaningful — training accuracy would read higher and tell you less.

Takeaways

The support vector machine is worth keeping in the kit even in a world that mostly reaches for gradient boosting. Its instinct — find the widest gap, and let only the points at the frontier define the boundary — produces classifiers that generalize cleanly and don't wobble when you perturb the easy examples. On a linearly separable or nearly-separable problem it's a genuinely strong, stable baseline, and the margin it reports tells you something logistic regression won't: how much room the boundary actually has.

Three things to carry forward. First, the margin as the objective: maximizing $2/\lVert w \rVert$ is the same as minimizing $\lVert w \rVert$, which is why the L2 term isn't an afterthought here the way it can be elsewhere — it is the margin. Second, SGD on the hinge loss is what makes the SVM scale; the elegant quadratic program is what you learn, but stochastic gradient descent on minibatches is what you run when the data is large, and it reaches the same place. Third, the choice between this and logistic regression comes down to what you need out of the model: reach for logistic regression when you want a calibrated probability and readable per-feature weights, reach for the SVM when you want the most stable separating boundary and you can live with a bare label. And when the boundary is genuinely curved, neither linear model can follow it — that's where the kernel trick comes in, swapping every dot product for a similarity function so the same margin-maximizing machinery draws nonlinear boundaries without ever leaving this math. We didn't implement it, but everything in this chapter is the linear special case of it, and that's the natural next step whenever a straight line isn't enough.