The perceptron

What this chapter covers

This is the first neuron in the book. Everything up to now has been statistics wearing a machine-learning hat: a threshold on one feature, a probability from a sigmoid, a line fit by descending a loss. The perceptron is the thing that got called a brain cell in 1958 and set off sixty years of neural networks. It's a single linear unit — a weighted sum, a hard threshold, a fire-or-don't output — trained by a rule so simple you can run it in your head: show it a point, and if it gets the point wrong, shove the weights toward the right answer. That's the whole algorithm.

We build it from scratch in NumPy: the net input, the step activation, the update rule, and an epoch loop that keeps sweeping the data until a full pass makes no mistakes. Then we let scikit-learn's Perceptron fit the same thing and check the numbers line up. The centerpiece is an animation of the learning rule itself — the decision line swinging and rotating across a 2-D scatter, one frame per correction, a mistake counter draining to zero underneath it.

And then the catch, because this chapter is really two stories. The perceptron converges in a finite number of steps if the data is linearly separable — a real theorem, provable, and we watch it happen. If the data isn't separable, it never settles; it oscillates forever, correcting the same points over and over. We demonstrate both. That second failure is not a footnote. It's the wall that stalled neural network research for over a decade, and it's exactly why the next chapter exists.

A bit of history

The idea starts with two people trying to model a brain with math. In 1943 Warren McCulloch, a neurophysiologist, and Walter Pitts, a logician, published a paper showing that a network of simple threshold units — sum your inputs, fire if you cross a threshold — could compute any logical function you wanted. No learning in it yet; you wired the weights by hand. But it was the first claim that thought might be arithmetic, that a neuron could be a logic gate, and it put the threshold unit on the table as a thing worth building.

Frank Rosenblatt built it. In 1958, a psychologist at Cornell, he took the McCulloch-Pitts unit and gave it a learning rule — a way to set the weights from examples instead of by hand — and called the result the perceptron. He wasn't modest about it. The Mark I Perceptron was actual hardware, a machine with a grid of photocells that learned to tell shapes apart, and the New York Times wrote it up in 1958 saying the Navy expected machines that would walk, talk, see, and reproduce themselves. That's the origin of the neural network hype cycle, and it reads exactly like the ones since.

Then in 1969 Marvin Minsky and Seymour Papert wrote a book, "Perceptrons," that did the math carefully and found the floor. A single perceptron can only draw a straight boundary, and there are dead-simple problems no straight boundary can solve — the exclusive-or, XOR, being the famous one. Two inputs, output true when exactly one is on. Four points, and no line separates them. Rosenblatt's machine couldn't learn it and never would. The book was read, fairly or not, as a verdict on the whole approach; funding dried up and the field went quiet for years, the stretch people now call the first AI winter. The irony is that the fix — stack the units in layers and the XOR problem falls apart — was already imaginable in 1969. What was missing was a way to train the stack, and that took until the 1980s. This chapter is the unit Minsky and Papert cornered; the next one is the escape.

The intuition

Forget the brain metaphor for a second, it does more harm than good. A perceptron is a line, plus a rule for which side is which. In two dimensions you have a scatter of points in two classes and you want a straight boundary with one class on each side. The perceptron holds a candidate boundary, checks each point, and when a point is on the wrong side it tilts and shifts the line to pull that point onto the correct side. Do that enough times and, if a perfect line exists, you land on one.

Here's the data we'll teach it on: 80 points, two classes, and the two blobs sit apart with clear space between them. I built this set on purpose so a straight line can separate it cleanly — that clean gap is the whole precondition for what follows.

The mechanism that makes this work is worth sitting with, because it's the germ of everything later. The unit doesn't fit anything in one shot and it doesn't minimize a loss. It reacts, one mistake at a time. A wrong point is a nudge in a direction; the weight vector accumulates those nudges; the line is wherever the accumulated nudges point. It's error-driven, it's online, and it's local — the same three properties that, generalized to many layers and a smooth activation, become backpropagation. Get this and you've got the shape of the whole field.

The math

Three equations, and the third is the only one that matters. Start with the net input. For a point $x_i$ with $D$ features, a weight vector $w$, and a bias $b$:

That's the same weighted sum from every linear model so far. The difference is what we do with it. Logistic regression squashed $z_i$ into a probability. The perceptron does the crudest possible thing — it looks at the sign. The activation is a hard step, and the prediction is which side of zero the score falls on:

This is why the labels live in $\{-1, +1\}$ instead of $\{0, 1\}$. It isn't a cosmetic choice; it's what makes the learning rule collapse to one line. The boundary is the flat surface where the score is exactly zero, $w^{\top} x + b = 0$, and everything on one side fires $+1$, everything on the other fires $-1$.

Now the rule. Take a point the unit got wrong — its prediction $\hat{y}_i$ disagrees with the truth $y_i$. Update the weights and bias like this, with a learning rate $\eta$:

Read what that does. If the true label is $+1$ and we missed it, we add the point to the weight vector, which increases $w^{\top}x_i$ next time and pushes the score toward positive. If the truth is $-1$, we subtract the point and push the score negative. Because $y_i$ is $\pm 1$, both cases are the one expression above. Correct points get no update at all — the perceptron only ever learns from its mistakes, which is either elegant or lazy depending on your mood.

One clean fact falls out and it's worth pocketing. Since we start the weights at zero, the learning rate $\eta$ only rescales $w$ and $b$ uniformly; it never changes which side of zero a score lands on, so it never changes a single prediction or the sequence of updates. On the plain perceptron, tuning the learning rate is busywork. I set it to 1 and move on.

And the theorem, because it's the reason any of this is guaranteed to work. If the two classes can be separated by some hyperplane with a margin — a strip of width $\gamma$ around the boundary with no points in it — then the number of updates the perceptron makes before it stops is at most $(R/\gamma)^2$, where $R$ is the radius of the smallest ball containing the data. Finite. Bounded. It does not depend on how many points you have or how many features. Rosenblatt's rule, dumb as it looks, is provably correct on separable data. The proof is short and the bound is the whole story: a bigger margin means fewer corrections, and no margin at all — non-separable data — means the bound is infinite, which is the polite way of saying it never stops.

What it's good at, what it isn't

I'll be honest about where this sits, because in 2026 you are not going to ship a raw perceptron. Its virtues are pedagogical and historical, not practical. What it's genuinely good at is being the simplest thing that learns: online, one point at a time, constant memory, no matrix to invert and no loss to differentiate. On data that really is linearly separable it finds a separating boundary and it finds it fast, and the convergence guarantee is the kind of clean result the rest of ML rarely gives you. As the atom that the mistake-driven, error-corrected style of learning is built from, it's the right place to start.

The problems are the reasons it's a history lesson. It stops at any separator, not the best one — the moment a pass is clean it quits, so the boundary it lands on can sit right up against a class instead of splitting the gap down the middle, which is exactly the flaw the support vector machine was invented to fix. On data that isn't separable it doesn't degrade gracefully; it never converges, thrashing between weight vectors forever with no signal that it's given up. And the deep limit is the one Minsky and Papert named: the boundary is a single straight line, so any problem whose classes aren't linearly separable is simply out of reach. XOR is the two-bit example, and most real problems are XOR with more dimensions and worse handwriting. You don't reach for this model in production. You reach for what it turned into.

The data

Two datasets, because this chapter has to show a success and a failure.

The first is the separable set above: 80 points, 31 in the positive class and 49 in the negative, drawn on either side of a fixed diagonal line with a gap of 0.35 carved out around it so no point sits in the margin. That gap is not decoration. It's what makes the data linearly separable by construction, which is what makes the convergence theorem apply. If I let points drift into the margin the guarantee would evaporate, so I reject them when I generate the data.

The second is XOR, the problem that broke the perceptron. Four Gaussian blobs at the corners of a square, labeled by a checkerboard: the two diagonal corners share a class, the other two share the other. Here it is.

Look at it and try to draw one straight line with cyan on one side and purple on the other. You can't, and it isn't a matter of trying harder — the two classes sit on opposite diagonals, so any line that gets one diagonal right hands the other diagonal to the wrong side. This is the whole of Minsky and Papert's argument in a picture. The perceptron will chase this data forever.

Build it, one function at a time

Four small pieces, bottom up, in the order you'd type them. Start with the net input — the weighted sum, the only arithmetic in the model.

def net_input(x, w, b):
    """The weighted sum plus bias for one sample: z = w·x + b.

    x and w are length-D vectors, b is a scalar. This is the whole linear
    part of the neuron — everything else is just deciding what to do with
    its sign.
    """
    return float(np.dot(w, x) + b)

Then the activation. This is the line that separates a perceptron from everything before it in the book: not a probability, not a real number, just a sign.

def step(z):
    """Step activation: fire +1 when the net input is non-negative, else -1.

    This hard threshold is what makes the perceptron a classifier rather
    than a regressor, and also what makes it non-differentiable — the
    reason later networks swap it out for a smooth activation.
    """
    return 1 if z >= 0.0 else -1


def predict(x, w, b):
    """Classify one sample by the sign of its net input."""
    return step(net_input(x, w, b))

I split step from predict on purpose. The step function is the activation — the thing later networks replace with something smooth — and keeping it as its own named piece makes that swap legible when we get there. Note the tie-break: a score of exactly zero fires $+1$. It's an arbitrary call on a measure-zero event, but pick one and be consistent.

Now the rule, the entire reason the perceptron learns. Everything above is scaffolding for these two lines.

def update(x, y, w, b, lr):
    """The perceptron correction for one misclassified sample.

    Push the boundary toward getting this point right: add the point to the
    weights if its true label is +1, subtract it if -1. Because y is ±1 the
    two cases fold into one line.

        w ← w + lr · y · x
        b ← b + lr · y

    Returns the updated (w, b). This is the entire learning rule.
    """
    w = w + lr * y * x
    b = b + lr * y
    return w, b

That's it. That's the algorithm the New York Times thought would walk and talk. Add the point to the weights, or subtract it, depending on its label. The bias moves by the label alone, which is the same rule with a constant feature of 1 folded in. If this looks too simple to work, that's the correct reaction, and it does work — on the right data.

Last piece, the loop that ties it together: sweep the data, update on every mistake, and stop when a whole pass is clean.

def fit(X, y, lr=1.0, max_epochs=50, seed=0, record=False):
    """Online perceptron learning: sweep the data, correct on every mistake.

    Start the weights at zero. One epoch is a full pass over the samples in
    a seeded random order. For each sample, if the unit's prediction
    disagrees with the truth, apply the perceptron update. Count the
    mistakes in the pass; if a whole epoch goes by with zero of them, the
    data is separated and we stop — that clean pass is the convergence
    check. With `record=True` we also return the per-update history and the
    mistakes-per-epoch list so the chapter can replay the learning.
    """
    rng = np.random.default_rng(seed)
    N, D = X.shape
    w = np.zeros(D)
    b = 0.0
    history = []
    mistakes_per_epoch = []
    for epoch in range(max_epochs):
        order = rng.permutation(N)
        mistakes = 0
        for i in order:
            xi, yi = X[i], y[i]
            if predict(xi, w, b) != yi:
                w, b = update(xi, yi, w, b, lr)
                mistakes += 1
                if record:
                    history.append((epoch, int(i), w.copy(), b, mistakes))
        mistakes_per_epoch.append(mistakes)
        if mistakes == 0:
            break  # a full pass with no corrections: the data is separated
    if record:
        return w, b, history, mistakes_per_epoch
    return w, b, mistakes_per_epoch

The structure carries the two ideas that matter. Each epoch shuffles the points with a seeded generator so the run is repeatable but the order isn't degenerate. And the stopping condition is the convergence check made literal: count the corrections in a pass, and if there were none, every point is on the right side — the data is separated and there is nothing left to do, so break. On XOR that condition is never met, which is why max_epochs exists as a mercy. The record flag stashes every correction so we can replay the learning frame by frame, which is the next section.

Watch it work

This is the perceptron learning, on the separable set so every point is visible. The top panel is the data; the orange line is the current decision boundary, the place where the score is zero. A red ring marks the point that triggered the correction in this frame — the one the unit just got wrong and is now fixing. The bottom panel counts the mistakes in each epoch; the bar for the pass we're inside grows as corrections happen.

Press play. Every frame is one real update out of fit — the exact weights the code produced, no smoothing, no re-ordering. Reset and run it again as many times as you want; it's the same sequence each time.

Watch the opening. The weights start at zero, so there's no line — the score is zero everywhere, the unit fires $+1$ for everything, and every negative point is a mistake waiting to be corrected. The first update creates a boundary out of nothing, crude and badly placed. Then the corrections start swinging it. Each red ring is a point on the wrong side, and each update rotates and slides the line to scoop that point up, sometimes knocking a different point out of place in the process. The line lurches rather than glides, which is the honest character of this rule — it fixes one point at a time and doesn't look ahead.

The bottom panel tells the same story as a countdown. Twelve mistakes in the first pass, then five, then six, then three, then zero. Notice it isn't monotonic — the third pass makes more mistakes than the second. That's real, and it's fine; the perceptron gives no guarantee that a given epoch improves on the last, only that on separable data the mistakes stop eventually. And they do: the whole thing converges after 5 passes and 26 total corrections. The final pass is the payoff — a clean sweep, no rings, no updates, every point on its correct side. That clean pass is the algorithm proving to itself that it's done.

The one thing to carry out of this animation: nothing here minimized a loss. There was no gradient, no downhill. There was a line reacting to its own mistakes until it ran out of them. That's a different idea of learning than the last few chapters, and it's the older one.

Now put the two datasets side by side and count mistakes per pass. This is the whole chapter in one chart. The separable set drops to zero and stops — the line ends where convergence kicked in and the loop broke. XOR never gets there; the count bounces around pass after pass, no clean sweep, no end, because there is no line to find.

That's the convergence theorem and its fine print in a single picture. Separable data: the mistakes hit zero in a handful of passes and the algorithm halts because it's provably done. Non-separable data: the same rule, the same code, thrashing forever with no way to tell it to stop except a hard epoch cap. The perceptron can't distinguish "still working" from "never going to finish," and that inability is the practical face of everything Minsky and Papert proved.

The full implementation

The whole file, top to bottom — net input, step, update, the training loop, and the accuracy helper the traces use. This is the code the animation actually ran:

"""The perceptron, built from scratch.

A single linear threshold unit — the original artificial neuron. Take a
weighted sum of the inputs plus a bias, and fire +1 or -1 depending on its
sign. Learning is the classic online perceptron rule: sweep the data one
point at a time, and every time the unit gets a point wrong, nudge the
weights toward the right answer. Repeat until a whole pass makes no
mistakes. On linearly separable data that pass is guaranteed to arrive
(the perceptron convergence theorem); on data that isn't, it never does.

Labels live in {-1, +1}, which is what makes the update rule so tidy.

Pure NumPy. Every function below appears in the chapter one step at a time;
the `# region:` markers are what the book's include directives pull in.
"""

import numpy as np


# region: net_input
def net_input(x, w, b):
    """The weighted sum plus bias for one sample: z = w·x + b.

    x and w are length-D vectors, b is a scalar. This is the whole linear
    part of the neuron — everything else is just deciding what to do with
    its sign.
    """
    return float(np.dot(w, x) + b)
# endregion


# region: step
def step(z):
    """Step activation: fire +1 when the net input is non-negative, else -1.

    This hard threshold is what makes the perceptron a classifier rather
    than a regressor, and also what makes it non-differentiable — the
    reason later networks swap it out for a smooth activation.
    """
    return 1 if z >= 0.0 else -1


def predict(x, w, b):
    """Classify one sample by the sign of its net input."""
    return step(net_input(x, w, b))
# endregion


# region: update
def update(x, y, w, b, lr):
    """The perceptron correction for one misclassified sample.

    Push the boundary toward getting this point right: add the point to the
    weights if its true label is +1, subtract it if -1. Because y is ±1 the
    two cases fold into one line.

        w ← w + lr · y · x
        b ← b + lr · y

    Returns the updated (w, b). This is the entire learning rule.
    """
    w = w + lr * y * x
    b = b + lr * y
    return w, b
# endregion


# region: fit
def fit(X, y, lr=1.0, max_epochs=50, seed=0, record=False):
    """Online perceptron learning: sweep the data, correct on every mistake.

    Start the weights at zero. One epoch is a full pass over the samples in
    a seeded random order. For each sample, if the unit's prediction
    disagrees with the truth, apply the perceptron update. Count the
    mistakes in the pass; if a whole epoch goes by with zero of them, the
    data is separated and we stop — that clean pass is the convergence
    check. With `record=True` we also return the per-update history and the
    mistakes-per-epoch list so the chapter can replay the learning.
    """
    rng = np.random.default_rng(seed)
    N, D = X.shape
    w = np.zeros(D)
    b = 0.0
    history = []
    mistakes_per_epoch = []
    for epoch in range(max_epochs):
        order = rng.permutation(N)
        mistakes = 0
        for i in order:
            xi, yi = X[i], y[i]
            if predict(xi, w, b) != yi:
                w, b = update(xi, yi, w, b, lr)
                mistakes += 1
                if record:
                    history.append((epoch, int(i), w.copy(), b, mistakes))
        mistakes_per_epoch.append(mistakes)
        if mistakes == 0:
            break  # a full pass with no corrections: the data is separated
    if record:
        return w, b, history, mistakes_per_epoch
    return w, b, mistakes_per_epoch
# endregion


def accuracy(X, y, w, b):
    """Fraction of samples the unit labels correctly."""
    preds = np.array([predict(x, w, b) for x in X])
    return float((preds == y).mean())

The library version

You wouldn't hand-roll this, and scikit-learn has it as Perceptron. It's the same unit and the same rule; the differences are ergonomic. It takes your labels in whatever form and handles the sign convention internally, it shuffles each epoch, and it wraps a few knobs the classic rule doesn't have — an early-stopping tolerance, an optional regularization penalty, a tunable learning rate. Turn the tolerance off and it runs the plain mistake-driven rule to convergence, which is what we want for an honest comparison:

def sklearn_perceptron(Xtr, ytr, Xte, yte, seed=0):
    """Fit sklearn's Perceptron and report its test accuracy.

    tol=None disables the early-stopping heuristic so it runs the classic
    rule to convergence, matching our from-scratch loop. Returns the
    learned weight vector, bias, test accuracy, and epochs taken (n_iter_).
    """
    clf = Perceptron(tol=None, max_iter=1000, shuffle=True, random_state=seed)
    clf.fit(Xtr, ytr)
    w = clf.coef_[0]
    b = float(clf.intercept_[0])
    acc = float(clf.score(Xte, yte))
    return w, b, acc, int(clf.n_iter_)

The one thing worth knowing is that Perceptron is a thin front on the same machinery as SGDClassifier — it's literally stochastic gradient descent on a particular loss (the hinge-like perceptron criterion) with the step activation. Which is a nice bridge: the reactive, error-driven rule we wrote by hand is also expressible as descent on a loss, and that second view is the one that scales to networks. Same algorithm, two ways of seeing it.

Scratch versus library

The face-off here isn't about a decimal point of accuracy, it's about a capability. The perceptron's promise is that it separates data it can separate, so I fit both models on each full dataset and asked whether they nailed it. On the separable set, both should hit 100%; on XOR, both should fail, and the failure is the point.

On the linearly separable data our from-scratch perceptron scores 1.0 and sklearn's scores 1.0 — both find a perfect separator, which is the convergence theorem cashed out as a number. They don't find the same separator; our weights came out around $[5.7, 8.0]$ with a bias of $-12$, sklearn's around $[3.3, 3.8]$ with a bias of $-6$, different lines through the same gap. That's the perceptron's indifference showing: any separator satisfies it, so two runs land on two different boundaries and both are, by the model's own standard, perfect. It's also the exact reason to prefer a max-margin method when generalization matters — the perceptron stops at the first line that works, not the best one.

On XOR the wheels come off, as they must. From scratch scores 0.525, sklearn scores 0.5125 — coin flips, both of them, on a balanced two-class problem. Neither implementation is broken; a straight line genuinely cannot do better here, so the better engineering in sklearn buys nothing. If you want the one chart that justifies the entire rest of a deep learning course, it's this one: two identical bars at chance on a problem a child solves by drawing two lines instead of one.

I'll flag the honest caveat. These are fit-set accuracies, not held-out — I'm measuring whether each model can separate the data it trained on, which is the right question for a capacity demonstration and the wrong one for generalization. On the separable set a held-out split would drop below 1.0, precisely because the perceptron picks an arbitrary separator that can sit close to one class. That gap between "found a line" and "found a good line" is real, and it's a big part of why the field kept going.

Takeaways

The perceptron is where neural networks start, and that's the reason to know it cold even though you'll never deploy one. Strip a modern network down to a single unit and this is what's left: a weighted sum, an activation, and a rule that corrects the weights from errors. Everything since is that idea scaled — more units, more layers, a smoother activation, a smarter update — but the skeleton is Rosenblatt's, and reading a backprop derivation is much easier once you've watched this line swing across a scatter.

Three things to carry forward. First, the convergence theorem is a genuine guarantee, and genuine guarantees are rare in this field — but it's guaranteed for a narrow world, linearly separable data, and it says nothing about the quality of the boundary or about generalization. A guarantee with fine print is still worth having, as long as you read the print. Second, the limitation is the lesson. The perceptron can only draw a straight line, XOR is not a straight-line problem, and no amount of training fixes that because it's a wall of representation, not of effort. When a model can't express the answer, you change the model, not the optimizer. Third, the two things holding it back point straight at the fix. The hard step activation has a flat, useless derivative, so you can't chain the rule through stacked units, and a single line can't bend. Swap the step for a differentiable activation and stack the units into layers, and both problems fall at once — the boundary can curve, and the errors can flow backward through the stack to train every layer. That is the multilayer perceptron, and it's the next chapter. XOR is where this one ends and where that one begins.