Support vector machines with SGD

Find the widest empty corridor between the two classes.

Only the points at the edge — the support vectors — define the boundary. Train it by stochastic gradient descent on the hinge loss.

Not a model of the data — a model of the boundary

Logistic regression asked every point for a probability.
The SVM asks: where's the widest gap I can run between the classes?
Only the points near the edge matter. Delete the rest — same boundary.
Those edge points are the support vectors. They are the model.

The data

Two blobs, 120 points, modest overlap. A straight line separates most; a few sit inside the corridor.

The margin, the hinge, the objective

Score and boundary, labels in {-1, +1}:

f(x) = w^{\top}x + b \qquad \text{margin width} = \frac{2}{\lVert w \rVert}

Wider margin means smaller $\lVert w \rVert$ . Pay a hinge for crowding it:

\ell_i = \max\big(0,\; 1 - y_i\, f(x_i)\big)

The soft-margin objective SGD minimizes — wide margin + few violations:

\mathcal{J} = \frac{\lambda}{2}\lVert w \rVert^{2} + \frac{1}{N}\sum_i \max\big(0,\, 1 - y_i f(x_i)\big)

$\lambda$ is the dial: wide margin vs. few points inside it.

Watch the margin settle

Each frame is one real epoch. Solid line = boundary; dashed = margins ( $f = \pm1$ ); amber rings = support vectors; red crosses = mistakes; hinge loss drops underneath.

Flat plane → huge margin → sweeps inward and settles. Support vectors collapse from all 120 to 33. Final hinge 0.156, margin width 1.27, 6 mistakes, 0.95 on the toy set.

Scratch vs. library vs. the exact solver

84 train / 36 test. Our SGD, sklearn's SGDClassifier(loss="hinge"), and the exact LinearSVC all land on the same boundary.

All three: 0.9444. Scratch $w = [1.31, 0.97]$ , sklearn $[1.30, 0.97]$ — same plane, two optimizers. 25 of 84 points are support vectors.

Takeaways

Maximize the margin: minimize $\lVert w \rVert$ . The L2 term is the margin.
Only support vectors define the boundary — a stable line, immune to the easy points.
SGD on the hinge is what makes the SVM scale; same answer as the quadratic program.
Logistic regression for a calibrated probability; SVM for the most stable boundary.
Straight line not enough? The kernel trick bends it — this is its linear special case.