Convolutional neural networks

What this chapter covers

The multilayer perceptron in the last chapter would happily take an image, but it takes it as a bag of pixels. Flatten an 8×8 digit into 64 numbers and every pixel gets its own weight, unrelated to its neighbors, and the network has no idea that pixel 9 sits directly below pixel 1. A convolutional network fixes exactly that. It bakes in what we already know about images — that a useful pattern is local, and that a pattern worth detecting in one corner is worth detecting everywhere — and it does it with one operation repeated across the image.

This chapter is about that one operation. We build convolution from scratch in NumPy: a filter sliding over an image, a ReLU that keeps the responses worth keeping, and max-pooling that shrinks the result while staying tolerant to small shifts. Then we watch it work, one pixel at a time, and put it to a small honest test — take scikit-learn's built-in handwritten digits, run a fixed bank of edge filters over them, and feed the resulting feature maps to a plain linear classifier. The same classifier on raw pixels is the control. Convolution wins, narrowly but really, and the point is to see why.

One thing up front, because the honesty matters. We are teaching the convolution mechanism, not training a deep network. The filters here are fixed edge detectors, not learned, and there's exactly one convolutional stage. A production CNN learns its filters end to end through backprop and stacks dozens of layers, and it does that in torch or tensorflow, not in a NumPy loop. What you'll build is the honest core those frameworks optimize.

A bit of history

The idea is older than the hardware that made it famous. In 1980 Kunihiko Fukushima published the Neocognitron, a layered network with local receptive fields and shared weights, explicitly inspired by Hubel and Wiesel's work on the cat visual cortex and its simple and complex cells. It had the whole architecture — local feature detectors, then a pooling step that tolerated position shifts — but no way to train it by gradient descent. It learned, but awkwardly.

Yann LeCun supplied the missing half. In 1989 at Bell Labs he trained a convolutional network with backpropagation to read handwritten ZIP codes, and by 1998 that line of work became LeNet-5, the network in "Gradient-based learning applied to document recognition" that banks actually deployed to read the numbers on checks. The convolution, the pooling, the trainable filters, the whole thing worked and shipped. Then it went quiet for over a decade, because it was expensive and the datasets were small.

2012 is when the field turned over. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered a deep convolutional network — AlexNet — in the ImageNet competition and cut the error rate by a margin that wasn't close. Same core idea LeCun had shipped in the nineties, now with GPUs, a million labeled images, ReLU activations, and dropout. Everything since is a descendant. The operation we're about to build by hand is the one that started all three chapters of that story.

The intuition

Think about what makes a digit a digit. It isn't the individual pixels; it's the strokes — an edge here, a curve there, a corner where two strokes meet. Those are local patterns, a few pixels wide, and where they appear matters less than that they appear. A 7 is a horizontal stroke on top of a diagonal one whether you write it high or low in the box.

A convolution is a small pattern detector you slide over the whole image. Take a tiny grid of weights — a filter, say 3×3 — line it up over a patch of the image, multiply overlapping cells, and add up the result into a single number. That number says how strongly this patch matches the pattern the filter is looking for. Slide the filter one pixel over and do it again. Sweep it across the whole image and you get a new image, a feature map, where each pixel reports the presence of that pattern at that spot. One filter, one pattern, detected everywhere, with the same handful of weights reused at every position.

That reuse is the whole trick, and it's two wins at once. First, a filter that's good at finding a vertical edge in the top-left is automatically good at finding one in the bottom-right, because it's the identical weights — you don't have to relearn the pattern for every location. Second, the parameter count collapses. A dense layer connecting 64 input pixels to 64 outputs needs 4,096 weights; a 3×3 filter needs 9, no matter how big the image gets. Here are five of the digits we'll work on, drawn as heatmaps — brighter is more ink:

Eight by eight is coarse — these are downsampled almost to abstraction — but the strokes are there, and a filter tuned to edges will light up right along them.

The math

A convolution — or cross-correlation, the version deep learning actually uses, which skips the kernel flip — is one double sum. Let $I$ be the input image and $K$ a kernel of height $k_h$ and width $k_w$. The output at position $(i, j)$ is the kernel laid over the patch anchored there, multiplied cell by cell and summed:

That's it — the receptive field is the patch $I(i \mathbin{:} i + k_h,\ j \mathbin{:} j + k_w)$, and $S(i, j)$ is one weighted sum of it. Slide over every position where the kernel fits fully inside the image (the "valid" mode) and an $H \times W$ image with a $k_h \times k_w$ kernel produces an $(H - k_h + 1) \times (W - k_w + 1)$ output. Our 8×8 digit and 3×3 filter give a 6×6 feature map.

The quiet, important part of that equation is what's not in it: $i$ and $j$ index the position, but $K$ does not. The same weights $K(m, n)$ act at every location. That is weight sharing, and it's the structural prior — the same feature detector everywhere, a parameter count set by the kernel size rather than the image size.

After the linear sum comes a nonlinearity, and for convolutional networks it's almost always the rectified linear unit, which keeps positive responses and zeros the rest:

An edge filter fires positive on one polarity of edge and negative on the other, so ReLU turns "this edge, this way, is here" into a clean signal and drops the mirror image into silence. Then pooling shrinks the map. Max-pooling over non-overlapping blocks of size $s$ keeps the strongest response in each block:

Pooling does two jobs. It reduces the number of features, and it buys a little translation tolerance — nudge an edge by a pixel inside a pooling block and the max is unchanged. Local detection, then a summary that doesn't care about the exact spot: that is the image prior, made of three equations.

What it's good at, what it isn't

Convolution is the right tool the moment your data has grid structure and local correlation — images most of all, but also audio spectrograms, time series, any signal where nearby samples relate and a pattern can appear anywhere. It's parameter-thrifty, because weight sharing means the filter count doesn't grow with the input size, so you can afford many filters and deep stacks. It's translation-tolerant by construction, thanks to pooling and the sliding itself. And it composes: stack convolutions and the early layers find edges, the middle ones find corners and textures out of those edges, the late ones find object parts. You get a hierarchy of features for free, learned from the data, which is why the same architecture reads digits, finds tumors, and captions photographs.

The costs are the mirror image of the assumptions. Convolution bets that the useful structure is local and translation-equivariant, and when that bet is wrong the prior hurts — tabular data with no spatial meaning gains nothing from pretending column 3 neighbors column 4. Real CNNs are also data-hungry and compute-hungry; the reason they slept from 1998 to 2012 is that they need lots of labeled examples and a GPU to train the filters, and none of that is free. And they're only translation-tolerant, not rotation- or scale-invariant — turn the image ninety degrees and a filter tuned to horizontal edges is looking for the wrong thing. In this chapter we sidestep the training cost entirely by fixing the filters instead of learning them, which is honest for teaching the operation and dishonest as a picture of what a deployed CNN does. Keep that seam in mind; we'll come back to it.

The data

One dataset, built into scikit-learn: load_digits, 1,797 images of handwritten digits at 8×8 resolution, 8 bits of grayscale squeezed down to integer intensities from 0 to 16. It's the small cousin of MNIST — same task, a tenth the resolution — and it's perfect here precisely because it's tiny. An 8×8 image is 64 numbers, small enough that a 3×3 filter sliding across it has just 36 stops, so we can animate every single one and still watch the feature map fill in by hand. No download, no preprocessing, seeded split; the whole thing runs in a second.

We use it two ways. For the concept animation we pick one image — the first 3 in the set — and slide a single edge filter over it, frame by frame. For the real result we run the full bank of filters over all 1,797 images, split 1,257 for training and 540 for testing (stratified, seed 0), and let a linear classifier sort them out. The labels only ever touch the classifier; the convolution never sees them.

Build it, one function at a time

Four small functions and two that assemble them, in the order you'd write them. The first is the convolution itself — the sliding weighted sum, exactly the double sum from the math section. An image and a kernel go in; for every position where the kernel fits, we pull out the patch, multiply by the kernel, and sum to one number:

def conv2d(image, kernel):
    """Valid 2-D cross-correlation: slide the kernel over the image.

    For every position where the kernel fits fully inside the image, multiply
    the overlapping patch by the kernel elementwise and sum. This is what deep
    learning calls "convolution" — technically cross-correlation, since we
    don't flip the kernel; the distinction doesn't matter when the filter is
    learned, and here it keeps the arithmetic readable.

    An (H, W) image and a (kh, kw) kernel give an (H-kh+1, W-kw+1) output.
    """
    H, W = image.shape
    kh, kw = kernel.shape
    out_h, out_w = H - kh + 1, W - kw + 1
    out = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            patch = image[i:i + kh, j:j + kw]      # the receptive field
            out[i, j] = np.sum(patch * kernel)     # weighted sum, one number
    return out

That's the entire operation. Two loops over output positions, a patch, an elementwise multiply, a sum. Everything else in a convolutional network is this function called many times with different weights. Next the nonlinearity, which is almost too small to write down but earns its place — it's what makes the edge responses one-sided:

def relu(x):
    """Rectified linear unit: keep positive responses, zero the rest.

    An edge filter fires positive on one polarity of edge and negative on the
    other. ReLU throws the negative half away, so each filter reports "how
    strongly this particular edge is present, and nowhere is it negatively
    present." To capture both polarities we pair every filter with its
    negation (see edge_filters); ReLU is what makes that pairing meaningful.
    """
    return np.maximum(x, 0.0)

Then pooling. Non-overlapping blocks, the max of each, a smaller map out. This is where the translation tolerance and the feature reduction both come from:

def max_pool2d(feature_map, size=2):
    """Downsample by taking the max over non-overlapping size x size blocks.

    Pooling buys two things at once: it shrinks the map (fewer features) and
    it makes the response tolerant to small shifts — if an edge moves by a
    pixel inside a pooling block, the max is unchanged. That translation
    tolerance is half of why convolution is the right prior for images.

    A (H, W) map with size s gives a (H//s, W//s) map; any ragged remainder
    on the right/bottom edge is dropped.
    """
    H, W = feature_map.shape
    out_h, out_w = H // size, W // size
    out = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            block = feature_map[i * size:(i + 1) * size,
                                j * size:(j + 1) * size]
            out[i, j] = block.max()
    return out

Now the filters. In a real CNN these are learned; here they're a fixed bank of eight edge detectors — the horizontal and vertical Sobel operators and the two diagonals, each paired with its own negation so that ReLU can capture both polarities of every edge. These are the classic hand-designed kernels that a trained network's first layer tends to rediscover on its own, which is why using them fixed is a fair stand-in for the mechanism:

def edge_filters():
    """A fixed bank of eight 3x3 edge detectors.

    Four oriented gradients — horizontal (Sobel), vertical (Sobel), and the
    two diagonals — each paired with its negation so ReLU can capture both
    polarities of every edge. Nothing here is learned; these are the classic
    hand-designed kernels a CNN's first layer tends to rediscover on its own.
    """
    sobel_h = np.array([[1, 0, -1], [2, 0, -2], [1, 0, -1]], dtype=float)
    sobel_v = np.array([[1, 2, 1], [0, 0, 0], [-1, -2, -1]], dtype=float)
    diag_a = np.array([[0, 1, 2], [-1, 0, 1], [-2, -1, 0]], dtype=float)
    diag_b = np.array([[2, 1, 0], [1, 0, -1], [0, -1, -2]], dtype=float)
    base = [("vertical edge", sobel_h), ("horizontal edge", sobel_v),
            ("diagonal /", diag_a), ("diagonal \\", diag_b)]
    bank = []
    for name, k in base:
        bank.append((name + " +", k))
        bank.append((name + " -", -k))
    return bank

With the pieces in hand, the per-image pipeline is a straight line: run each filter, rectify, pool, flatten, and concatenate. Eight filters give eight 6×6 maps; 2×2 pooling shrinks each to 3×3; flattened and stacked that's $8 \times 3 \times 3 = 72$ numbers per image:

def extract_features(image, filters, pool=2):
    """Turn one image into a feature vector: conv -> ReLU -> pool, per filter.

    Run every filter over the image, rectify, pool down, flatten, and
    concatenate. Eight 3x3 filters over an 8x8 image give eight 6x6 maps;
    2x2 pooling shrinks each to 3x3, so the whole image collapses to a
    8 * 3 * 3 = 72-dimensional vector — fewer numbers than the 64 raw pixels,
    but each one summarizes an oriented edge in a small region.
    """
    parts = []
    for _name, kernel in filters:
        conv = conv2d(image, kernel)
        pooled = max_pool2d(relu(conv), pool)
        parts.append(pooled.ravel())
    return np.concatenate(parts)

And finally the batch wrapper — the same extraction over every image, producing the design matrix the classifier trains on. The convolution is a fixed, untrained front end; all the actual learning happens downstream in the linear head:

def conv_features(images, filters=None, pool=2):
    """Stack extract_features over a batch of images into a design matrix.

    images is (N, H, W); the result is (N, F) with F the per-image feature
    count. This matrix is exactly what the linear classifier trains on — the
    convolution is a fixed, untrained front end, and all the learning happens
    in the linear head downstream.
    """
    if filters is None:
        filters = edge_filters()
    return np.array([extract_features(img, filters, pool) for img in images])

Seventy-two features out of sixty-four pixels in. It's barely a compression, but every one of those numbers means something — the strength of a particular oriented edge in a particular region — where a raw pixel means only "how bright is this one spot."

Watch it work

Here is the operation itself, slowed down until you can read it. The animation below runs the real conv2d on one digit — a handwritten 3 — with the vertical edge filter, the Sobel kernel that responds to left-to-right changes in brightness. On the left is the input image, 8×8, with the current 3×3 receptive field outlined in orange. On the right is the feature map filling in, one pixel per frame, with the cell being written outlined to match. The caption names the position and the exact number the filter computed for that patch.

Follow the orange box. It starts on the top-left patch, the filter multiplies and sums to one number, that number lands in the top-left of the output, and then the box steps right. Thirty-six stops later — six across, six down — the feature map is complete, and it's a different picture than the input: bright where the digit has a vertical edge, dark where it's flat or where the edge runs the other way. Watch the values in the caption go positive on the left flank of a stroke and negative on the right, the two sides of the same edge. That sign is exactly what ReLU will act on next, keeping the positive flank and zeroing the negative one. Reset and run it again; it's fully deterministic, the same sweep every time, because convolution is just arithmetic — no seed, no randomness, one weighted sum per position.

Now the same operation with all eight filters at once. This is one digit — the same 3 — pushed through the full bank, each panel a filter's response after ReLU. Different filters light up on different strokes:

Each filter is a specialist. The vertical-edge pair fires on the up-and-down strokes, the horizontal pair on the flat ones, the diagonals on the slants, and each pair's two halves light up on opposite edges of the same stroke because one is the other's negation. Stack these eight small views and you have a richer description of the digit than the raw pixels ever gave — not "how bright is spot 37" but "there's a vertical edge here and a diagonal there." That's the representation the classifier gets.

The full implementation

The whole convolution front end, no framework, top to bottom. This is exactly what the animation ran and what the accuracy numbers below come from:

"""Convolutional feature extraction, built from scratch.

The convolution machinery of a CNN — a filter sliding over an image, a ReLU
that keeps the positive responses, and max-pooling that shrinks the map while
keeping the strongest activation in each patch — written in pure NumPy with no
autodiff and no deep-learning framework anywhere.

The pipeline is: for each small edge-detecting filter, slide it over the image
(valid cross-correlation), rectify the response, pool it down, then flatten and
concatenate every filter's pooled map into one feature vector. That vector is
what a plain linear classifier gets trained on. We do NOT train the filters
here — they're fixed edge detectors — and we do NOT stack conv layers. This is
the convolution *operation* taught honestly, not a production training loop.

Every function below appears in the chapter one step at a time; the
`# region:` markers are what the book's include directives pull in.
"""

import numpy as np


# region: conv2d
def conv2d(image, kernel):
    """Valid 2-D cross-correlation: slide the kernel over the image.

    For every position where the kernel fits fully inside the image, multiply
    the overlapping patch by the kernel elementwise and sum. This is what deep
    learning calls "convolution" — technically cross-correlation, since we
    don't flip the kernel; the distinction doesn't matter when the filter is
    learned, and here it keeps the arithmetic readable.

    An (H, W) image and a (kh, kw) kernel give an (H-kh+1, W-kw+1) output.
    """
    H, W = image.shape
    kh, kw = kernel.shape
    out_h, out_w = H - kh + 1, W - kw + 1
    out = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            patch = image[i:i + kh, j:j + kw]      # the receptive field
            out[i, j] = np.sum(patch * kernel)     # weighted sum, one number
    return out
# endregion


# region: relu
def relu(x):
    """Rectified linear unit: keep positive responses, zero the rest.

    An edge filter fires positive on one polarity of edge and negative on the
    other. ReLU throws the negative half away, so each filter reports "how
    strongly this particular edge is present, and nowhere is it negatively
    present." To capture both polarities we pair every filter with its
    negation (see edge_filters); ReLU is what makes that pairing meaningful.
    """
    return np.maximum(x, 0.0)
# endregion


# region: max_pool2d
def max_pool2d(feature_map, size=2):
    """Downsample by taking the max over non-overlapping size x size blocks.

    Pooling buys two things at once: it shrinks the map (fewer features) and
    it makes the response tolerant to small shifts — if an edge moves by a
    pixel inside a pooling block, the max is unchanged. That translation
    tolerance is half of why convolution is the right prior for images.

    A (H, W) map with size s gives a (H//s, W//s) map; any ragged remainder
    on the right/bottom edge is dropped.
    """
    H, W = feature_map.shape
    out_h, out_w = H // size, W // size
    out = np.zeros((out_h, out_w))
    for i in range(out_h):
        for j in range(out_w):
            block = feature_map[i * size:(i + 1) * size,
                                j * size:(j + 1) * size]
            out[i, j] = block.max()
    return out
# endregion


# region: edge_filters
def edge_filters():
    """A fixed bank of eight 3x3 edge detectors.

    Four oriented gradients — horizontal (Sobel), vertical (Sobel), and the
    two diagonals — each paired with its negation so ReLU can capture both
    polarities of every edge. Nothing here is learned; these are the classic
    hand-designed kernels a CNN's first layer tends to rediscover on its own.
    """
    sobel_h = np.array([[1, 0, -1], [2, 0, -2], [1, 0, -1]], dtype=float)
    sobel_v = np.array([[1, 2, 1], [0, 0, 0], [-1, -2, -1]], dtype=float)
    diag_a = np.array([[0, 1, 2], [-1, 0, 1], [-2, -1, 0]], dtype=float)
    diag_b = np.array([[2, 1, 0], [1, 0, -1], [0, -1, -2]], dtype=float)
    base = [("vertical edge", sobel_h), ("horizontal edge", sobel_v),
            ("diagonal /", diag_a), ("diagonal \\", diag_b)]
    bank = []
    for name, k in base:
        bank.append((name + " +", k))
        bank.append((name + " -", -k))
    return bank
# endregion


# region: extract_features
def extract_features(image, filters, pool=2):
    """Turn one image into a feature vector: conv -> ReLU -> pool, per filter.

    Run every filter over the image, rectify, pool down, flatten, and
    concatenate. Eight 3x3 filters over an 8x8 image give eight 6x6 maps;
    2x2 pooling shrinks each to 3x3, so the whole image collapses to a
    8 * 3 * 3 = 72-dimensional vector — fewer numbers than the 64 raw pixels,
    but each one summarizes an oriented edge in a small region.
    """
    parts = []
    for _name, kernel in filters:
        conv = conv2d(image, kernel)
        pooled = max_pool2d(relu(conv), pool)
        parts.append(pooled.ravel())
    return np.concatenate(parts)
# endregion


# region: conv_features
def conv_features(images, filters=None, pool=2):
    """Stack extract_features over a batch of images into a design matrix.

    images is (N, H, W); the result is (N, F) with F the per-image feature
    count. This matrix is exactly what the linear classifier trains on — the
    convolution is a fixed, untrained front end, and all the learning happens
    in the linear head downstream.
    """
    if filters is None:
        filters = edge_filters()
    return np.array([extract_features(img, filters, pool) for img in images])
# endregion

The library version

There's no from-scratch classifier here, on purpose. The point of the experiment is to change the representation and hold the model fixed, so both sides of the face-off use the same scikit-learn linear classifier — multinomial logistic regression — and the only difference is what goes in. Standardize the features, fit, score:

def linear_head(X_train, y_train, X_test, y_test, seed=0):
    """Standardize the features, fit multinomial logistic regression, score.

    StandardScaler is fit on the training features only, then applied to both
    splits — the usual guard against leaking test statistics into training.
    LogisticRegression with the lbfgs solver is deterministic given the data,
    but we pass a seed anyway so the intent is explicit.

    Returns (test_accuracy, n_features).
    """
    scaler = StandardScaler().fit(X_train)
    Xtr = scaler.transform(X_train)
    Xte = scaler.transform(X_test)
    clf = LogisticRegression(max_iter=5000, C=1.0, random_state=seed)
    clf.fit(Xtr, y_train)
    acc = float(clf.score(Xte, y_test))
    return acc, int(X_train.shape[1])

That's the honest test of whether convolution helps. Feed this function the 72 convolutional features and you get one accuracy; feed it the 64 raw pixels and you get another; the classifier, the solver, the regularization, the seed are all identical. Any gap is the representation and nothing else. This is also where the scope line sits: a real CNN would replace both the fixed filters and this linear head with layers trained together by backprop in torch or tensorflow. We train only the head, on top of frozen filters, which is enough to show the prior earning its keep and far short of the real training loop.

Scratch versus library

Same digits, same classifier, same split — 1,257 images to train, 540 to test — and the only thing that changes is whether the classifier sees raw pixels or convolutional features:

Convolution features land at 0.9796 on the held-out 540 images; the same classifier on raw pixels gets 0.9722. It's a narrow win — about four extra correct digits out of 540 — and I want to be careful about how much weight it carries. On a dataset this small and this clean, raw-pixel logistic regression is already strong, so there isn't much room above it, and the honest reading is "the fixed edge features are competitive with raw pixels, and slightly better," not "convolution triples your accuracy." The gap would widen with harder images, learned filters, and more than one conv layer; here it's a hint, not a landslide.

The number that better shows what's happening under the hood is ReLU's discard rate. Across every training image and all eight filters, 52.0% of the convolution responses come out zero or negative and get thrown away. Roughly half of every feature map is silence — the filter looked, found no edge of its polarity, and reported nothing. That's not waste; it's the point. Each surviving activation is a confident "yes, this specific edge is here," and the classifier gets a sparse, oriented description instead of a dense wall of pixel intensities. Fewer features worth more each.

Takeaways

Convolution is what you reach for when the data has a grid and the meaningful patterns are local and could show up anywhere — images first, but audio, video, and many time series too. The reason it works isn't mysterious once you've built it: it's a prior, a hard-coded assumption that a good feature detector is small and that a pattern worth finding in one place is worth finding in all of them. Weight sharing turns that assumption into a tiny parameter count, pooling turns it into tolerance for small shifts, and stacking turns simple edge detectors into detectors for whatever the data is made of. When those assumptions match your problem, a CNN is enormously more sample-efficient than a dense network that has to learn the same feature separately at every location; when they don't — plain tabular data, no spatial meaning — the prior is dead weight and you should reach for something else.

What this chapter did not do is the part that makes CNNs actually powerful. We used fixed edge filters and one convolutional stage feeding a linear model, because that isolates the operation and lets you watch it. A real convolutional network learns its filters end to end by backpropagation — the same gradient machinery from the perceptron chapter, now flowing through convolutions and pooling — and stacks many layers so the features compound into a hierarchy. That training loop lives in torch or tensorflow, on a GPU, on far more data than 1,797 tiny digits, and it's a deliberate non-goal here. The mechanism you built by hand is the honest core of it: the sliding filter, the rectified response, the pooled summary. Everything a deep CNN adds is more of these, learned instead of fixed, which is the best possible reason to have run the fixed one by hand first.