Convolutional neural networks

A small filter, slid over an image, detects a local pattern everywhere at once.

Weight sharing + pooling = the right prior for grid data. Built from scratch in NumPy; the mechanism, not a training loop.

The idea

An MLP takes an image as a bag of pixels — no notion that pixel 9 sits below pixel 1. Convolution bakes in what we know: patterns are local, and a pattern worth finding in one corner is worth finding everywhere.

One filter = one small pattern detector, reused at every position. 9 weights, not 4,096, and the count never grows with the image.

The data

load_digits — 1,797 handwritten digits, 8×8, intensities 0–16. MNIST's tiny cousin. Small enough to animate every one of a filter's 36 stops.

The math

The sliding weighted sum, one output pixel:

S(i, j) = \sum_{m=0}^{k_h - 1} \sum_{n=0}^{k_w - 1} I(i + m,\ j + n)\, K(m, n)

Rectify, then pool the strongest response per block:

\mathrm{ReLU}(x) = \max(0,\ x)

P(i, j) = \max_{0 \le a, b < s} S(i \cdot s + a,\ j \cdot s + b)

The kernel $K$ has no $i, j$ index — same weights everywhere. That's weight sharing, the whole prior.

Watch it work

One edge filter slides over a handwritten 3. The receptive field is outlined on the input; the feature map fills one pixel per frame; the caption names the computed value.

Values go positive on one flank of a stroke, negative on the other — the two sides of an edge. ReLU keeps one, zeros the other.

Eight filters, one digit

Each filter is a specialist — vertical, horizontal, the two diagonals, each with its negation. The ReLU feature maps for one 3:

Scratch vs. library

Same classifier, same split (1,257 train / 540 test). Only the representation changes: 72 conv features vs. 64 raw pixels.

Conv features 0.9796 vs. raw pixels 0.9722 — a narrow, real win. ReLU discards 52% of responses: sparse, oriented features worth more each.

Takeaways

Convolution is a prior: local patterns, detected everywhere with shared weights.
Weight sharing → tiny parameter count; pooling → tolerance to small shifts.
Right for grids (images, audio, some time series); dead weight for plain tables.
This taught the mechanism, not training — fixed filters, one conv stage.
Real CNNs learn filters end to end and stack deep, in torch/tf on a GPU.