Logistic regression

Keep the straight boundary. Let every feature vote. Hand back a probability.

The default linear classifier — and the model you'll actually ship.

The move up from a threshold

Week 1: one feature, one vertical line, a hard yes/no.
Now: every feature votes, the line can tilt, the answer is a probability.
0.5 is still the natural cutoff — but now it's a dial you can move.
Calibrated probabilities + readable weights + a strong baseline. Hard to beat for the price.

Two classes, some overlap. A straight line can mostly separate them — but it has to tilt, because both features carry signal.

Linear score, then squash it to a probability:

z = w^{\top}x + b \qquad p = \sigma(z) = \frac{1}{1 + e^{-z}}

Train by minimizing cross-entropy — the loss the sigmoid was built for:

\mathcal{L} = -\frac{1}{N}\sum_i \big[\, y_i \log p_i + (1 - y_i)\log(1 - p_i)\,\big]

The gradient collapses to the error times the features:

\frac{\partial \mathcal{L}}{\partial w} = \frac{1}{N}\sum_i (p_i - y_i)\,x_i

Clean (p - y). That cancellation is why you don't use squared error here.

Each frame is one real gradient step. The orange line is where p = 0.5; red rings are current mistakes; the loss drops underneath.

Zero weights → no line → a crude cut → rotate and slide into the gap. 96% on the toy set.

Flat at the ends (confident), steep in the middle (uncertain), crossing 0.5 at z = 0.

Pima diabetes, 537 train / 231 test. Our gradient descent lands on sklearn's answer; both beat the one-feature threshold.

Scratch 0.7662 · sklearn 0.7662 · week-1 stump 0.7316. Heaviest weights: glucose, BMI, pregnancies — in that order.

The default linear classifier. The baseline a heavier model has to actually beat.
Probabilities, not labels: set the threshold to fit the cost of each mistake.
Standardize, then read the weights — they rank your signal for free.
Sigmoid + cross-entropy are a pair; the gradient cancels to (p - y). Don't use squared error.
The engine under the neural nets later — a network is this model stacked and bent until the boundary can curve.