Multilayer perceptrons and backpropagation

One hidden layer, one nonlinearity, and the wall the perceptron hit falls down.

The escape from XOR — depth plus nonlinearity, trained by the chain rule run backward.

Where it comes from

The fix (stack the layers) was obvious in 1969. Training the stack was not.
Backprop: Werbos 1974, rediscovered a few times, made to land by Rumelhart, Hinton & Williams, Nature 1986 — a hidden layer learning useful representations.
Cybenko 1989: one hidden layer can approximate any continuous function. Hornik 1991: it's the nonlinearity that does it, not the specific activation.

Existence, not a recipe — but it settled the question Minsky and Papert raised.

The idea

A perceptron computes one weighted sum: one line. A hidden layer computes several at once, and the output layer draws a line through those — a bent shape back in the original space.

The nonlinearity is not optional. Two stacked linear layers collapse into one line (a product of matrices is a matrix). The tanh in the middle breaks the collapse. Depth without nonlinearity buys nothing; depth with it buys everything.

The hidden layer learns a new set of coordinates where the problem is linearly separable, then separates it there.

The moons

Two interleaving crescents, 200 points, no straight line comes close.

The whole algorithm

Forward: two linear layers, a tanh between, a sigmoid on top.

Z_1 = X W_1 + b_1 \qquad A_1 = \tanh(Z_1) \qquad \hat{y} = \sigma(A_1 W_2 + b_2)

Loss: binary cross-entropy — the output unit is a logistic unit.

L = -\tfrac{1}{N} \sum_i \big[ y_i \log \hat{y}_i + (1 - y_i)\log(1 - \hat{y}_i) \big]

Backprop: the chain rule, output first, then push back through the weights.

\delta_2 = \hat{y} - y \qquad \delta_1 = (\delta_2 W_2^{\top}) \odot (1 - A_1^2)

Step downhill and repeat.

W \leftarrow W - \eta\, \tfrac{\partial L}{\partial W}

Watch the boundary bend

Random line → kink → bend → a curve wrapping the crescents. Real weights per frame; loss falling underneath.

Epoch 0 is a badly-placed straight cut. By the end: a curve a single line could never draw, train accuracy ~0.96, loss ~0.07.

The payoff: XOR

The network carves an X — two bent lines — through the four points. Exactly the shape XOR needs and a perceptron cannot make.

Scratch vs library

XOR: both hit 1.0 — the perceptron scored 0.5. The layer unlocked it. Moons (60 test points): scratch 0.983, sklearn 0.867.

Ours fits harder (full-batch, 4000 epochs); sklearn regularizes and early-stops. Both draw a curve — they just draw slightly different ones. That's nonconvexity.

Takeaways

Depth + nonlinearity is the whole jump. Without both you have a perceptron.
Backprop is the chain rule reusing the forward pass's cached activations — the whole gradient costs about one extra forward pass.
The guarantees are gone: nonconvex loss, so init, learning rate, and stopping all move the answer. You manage it by watching the loss curve.
This is the foundation under the CNN chapter — same forward-loss-backward-update loop, just swap the dense layer for convolutions.