Normalization and standardization

What this chapter covers

Here's a problem no algorithm in this book solves for you, because it happens before the algorithm ever runs. Your data has a glucose column that lives in the hundreds and a diabetes-pedigree column that lives between zero and one. To a model that measures distance, those two columns are not equal citizens — glucose is a hundred times louder simply because its numbers are bigger, and the pedigree column, which might carry real signal, gets drowned out. Nobody decided that. It's an accident of units, and it quietly wrecks any method that adds features together or measures how far apart two points are.

The fix is to put every feature on a common scale first. This chapter builds the three scalers you'll actually use — min-max normalization, standardization, and robust scaling — from scratch in NumPy, each one a fit/transform pair. Then we let scikit-learn build the same three and confirm they land on identical numbers to floating-point precision. The point of the chapter isn't any one formula; it's the discipline around them: you fit the scaler on the training set only, you apply those frozen statistics to the test set, and you never let the test data's own distribution leak into the transform. Get that wrong and your held-out score is fiction.

The dataset is the Pima diabetes set, chosen precisely because its features are on wildly different scales. And to see why scaling matters, we watch a distance-based method — a k-nearest-neighbors vote — change its answer as we rescale the axes, without touching a single data point.

A bit of history

Standardization isn't a machine-learning invention. It falls straight out of the oldest idea in statistics: the bell curve and the number of standard deviations you sit from its center. Abraham de Moivre wrote down the normal curve in 1733 as an approximation to coin-flip counts. Gauss and Laplace formalized it in the early 1800s around the theory of measurement error. But the word we lean on came later — Karl Pearson coined "standard deviation" in 1893, giving a single name to the spread of a distribution, and with it the natural move of measuring any value in units of that spread. Subtract the mean, divide by the standard deviation, and you get the z-score: how many standard deviations above or below average this point is. That's standardization, and it predates computers by a century. When we standardize a feature we're just applying the z-score column by column, so a value of +2 means the same thing — two standard deviations high — whether the column is glucose or age.

Robust scaling comes from a later and more skeptical tradition. John Tukey, in his 1977 book on exploratory data analysis, argued that the mean and standard deviation are too easily yanked around by a few extreme values, and pushed the median and the interquartile range — the boxplot's box — as summaries that a handful of outliers can't move. Robust scaling is that argument turned into a transform: center on the median, scale by the IQR. Min-max normalization has no single famous parent; it's a practitioner's habit from numerical computing and early neural networks, where you wanted every input pinned into a fixed [0, 1] box so nothing overflowed and nothing dominated. Three transforms, three different eras of the same instinct: make the numbers comparable before you trust a distance.

The intuition

Distance doesn't care about meaning; it cares about magnitude. Take two people in the Pima data who differ by 30 units of glucose and by 0.5 in pedigree. Squared-distance-wise, the glucose gap contributes 900 and the pedigree gap contributes 0.25. The pedigree difference is more than three thousand times quieter — not because it matters less, but because its numbers are smaller. Run k-nearest-neighbors on the raw columns and "nearest" almost entirely means "similar glucose." You've built a one-feature model by accident and called it eight-dimensional.

Scaling removes the accident. If you re-express every column in its own units of spread — standard deviations, or fractions of its range — then a one-unit move means the same amount of "surprise" in every column, and distance finally weighs all the features on their merits. Nothing about the data changes; the points don't move in any meaningful sense. What changes is the ruler. That's the whole idea, and it's easiest to see on two features with a big scale gap:

Each row is one feature's spread — the thin line runs min to max, the bar is the middle half (the interquartile range), the dark tick is the median. Look at what a shared axis does to them. Insulin and glucose stretch across the whole width; pregnancies, BMI, and the pedigree column are squashed into a sliver near zero, their real variation invisible. A distance metric reads this chart literally. It's going to listen to the long bars and ignore the short ones.

The math

Three transforms, one line each. In all of them $x$ is a single feature column and $x'$ (or $z$) is that column after scaling; every statistic is computed down the column, over the rows you fit on.

Min-max normalization slides and squeezes the column so its smallest value becomes 0 and its largest becomes 1:

Here $x_{\min}$ and $x_{\max}$ are the column's minimum and maximum. Every scaled value lands in $[0, 1]$, with the range fixed by the two most extreme points — which is exactly why a single outlier can wreck it.

Standardization is the z-score: recenter on the mean, rescale by the standard deviation:

$\mu$ is the column mean and $\sigma$ its standard deviation (population form, dividing by $N$). The output has mean 0 and standard deviation 1, and it isn't bounded — a genuine outlier stays far out at $z = 6$ instead of being folded into a fixed box.

Robust scaling keeps the z-score's shape but swaps in statistics that outliers can't drag around:

where $\operatorname{IQR}(x) = Q_3 - Q_1$ is the interquartile range, the gap between the 75th and 25th percentiles. The median and the IQR describe the bulk of the data and shrug off the tails, so a few absurd readings — an insulin of zero standing in for a missing value, say — don't distort the scale for everyone else.

What it's good at, what it isn't

The honest framing here isn't advantages and disadvantages of scaling; it's which models need it and which don't care. Anything that measures distance or adds features together needs it: k-nearest-neighbors, k-means, SVMs with an RBF kernel, PCA, anything with a Euclidean or dot-product core. Anything trained by gradient descent needs it too, because features on lopsided scales give the loss surface stretched, canyon-like contours that gradient descent zig-zags down slowly; scaling rounds the bowl out and the optimizer converges faster. And any model with an L1 or L2 penalty needs it, because the penalty is applied per coefficient and a feature measured in thousands gets a systematically different squeeze than one measured in fractions. For all of these, unscaled features aren't just suboptimal — they quietly change what the model is fitting.

Then there's the other camp, which doesn't care at all. Decision trees, random forests, and gradient-boosted trees split one feature at a time on thresholds, and a threshold at glucose > 140 means the same thing whether glucose is stored in the hundreds or rescaled to 0.7. Any monotonic rescaling leaves every possible split intact, so scaling a tree's inputs changes nothing but wastes your afternoon. That's the quick rule I carry: if the model thinks in distances, gradients, or penalties, scale; if it thinks in thresholds, don't bother. Even among the scalers the choice matters — min-max when you need a hard bounded range, standardization as the sane default, robust when the column is full of outliers you can't clean.

The data

The Pima Indians diabetes dataset: 768 patients, eight medical features, and a binary outcome for whether each was diagnosed with diabetes. It's a favorite for exactly the reason we want it here — the features are on comically different scales. Glucose and blood pressure sit in the tens and hundreds, insulin ranges into the hundreds, BMI in the tens, age in years, pregnancies in single digits, and the diabetes-pedigree function is a synthesized score between roughly 0 and 2.4. The raw-spread chart above is this dataset. On the training split the glucose column has a standard deviation about 95 times larger than the pedigree column's, which is the scale gap the whole chapter is built around.

We split it once — 576 train, 192 test, stratified on the outcome, seeded — and that split is fixed for every number on this page. Now the same eight features after standardization, fit on the full set for illustration:

Same eight features, same data, one shared axis — and now every box sits over zero with a comparable width. The middle halves line up; the differences that remain are the long tails, where you can read off which features are skewed (insulin and pedigree still trail far to the right, which is the argument for robust scaling on those two). This is the "everything is now comparable" picture. A distance metric handed these columns weighs all eight on equal terms.

Build it, one function at a time

Each scaler is two small functions: a fit that reads a statistic off the data and a transform that uses it. Splitting them this way isn't ceremony — it's the mechanism that lets you fit on train and apply to test, which is the only part of this chapter you can actually get wrong.

Min-max first. Fitting means recording each column's min and max; transforming means sliding the min to zero and dividing by the span:

def minmax_fit(X):
    """Learn the per-column min and max — the range of each feature.

    X is (N, D). Returns the two length-D vectors the transform needs. This is
    the only thing min-max "learns", and it learns it from whatever rows you
    hand it — so hand it the training set, never the whole dataset.
    """
    return {"min": X.min(axis=0), "max": X.max(axis=0)}


def minmax_transform(X, stats):
    """Squeeze every column into [0, 1] using a fitted range.

    Subtract the min so the smallest value maps to 0, divide by the span so the
    largest maps to 1. A column with no spread (max == min) would divide by
    zero, so we floor the denominator at 1 and leave that column at 0.
    """
    span = stats["max"] - stats["min"]
    span = np.where(span == 0, 1.0, span)
    return (X - stats["min"]) / span

The only defensive move is flooring the span at 1 for a constant column, so a feature with no spread doesn't divide by zero. Standardization has the same shape — fit records the mean and standard deviation, transform computes the z-score:

def standardize_fit(X):
    """Learn the per-column mean and standard deviation.

    std uses ddof=0 (the population formula, dividing by N) to match
    scikit-learn's StandardScaler exactly. A zero-variance column gets its
    scale floored at 1 so the transform doesn't divide by zero.
    """
    std = X.std(axis=0)
    std = np.where(std == 0, 1.0, std)
    return {"mean": X.mean(axis=0), "std": std}


def standardize_transform(X, stats):
    """The z-score: subtract the mean, divide by the standard deviation.

    Every column comes out centered at 0 with unit spread, so a value of +2
    means "two standard deviations above this feature's mean" no matter which
    feature it is. That shared meaning is what makes columns comparable.
    """
    return (X - stats["mean"]) / stats["std"]

I use ddof=0 deliberately, the population standard deviation dividing by $N$, because that's what scikit-learn's StandardScaler does and I want the two implementations to match to the last digit later. Robust scaling swaps the mean and std for the median and IQR:

def robust_fit(X):
    """Learn the per-column median and interquartile range (IQR).

    The median is the 50th percentile; the IQR is the 75th minus the 25th, the
    width of the middle half of the data. Neither one moves when a handful of
    points fly off to extreme values, which is what "robust" means here.
    np.percentile uses linear interpolation, matching sklearn's RobustScaler.
    """
    q25, q50, q75 = np.percentile(X, [25, 50, 75], axis=0)
    iqr = q75 - q25
    iqr = np.where(iqr == 0, 1.0, iqr)
    return {"median": q50, "iqr": iqr}


def robust_transform(X, stats):
    """Center on the median, scale by the IQR.

    Same shape as the z-score, but built from statistics that outliers can't
    drag around. The output isn't centered at exactly 0 mean or unit variance;
    it's centered at the median with the middle 50% spanning one unit.
    """
    return (X - stats["median"]) / stats["iqr"]

np.percentile with its default linear interpolation is the same rule sklearn's RobustScaler uses, so again the numbers will agree. Three transforms, all the same skeleton, differing only in which pair of statistics they read.

Now the function the whole chapter exists to justify. Given a train and a test split and any fit/transform pair, it fits on train alone and applies those frozen statistics to both:

def scale_train_test(Xtr, Xte, fit, transform):
    """Fit a scaler on TRAIN, apply it to both splits — no leakage.

    fit sees only Xtr, so the statistics (min/max, mean/std, median/IQR) are
    computed from training rows alone. The exact same fitted numbers rescale
    the test rows. If you instead fit on the full dataset, the test set's own
    distribution bleeds into the transform and your held-out score is a lie.
    """
    stats = fit(Xtr)
    return transform(Xtr, stats), transform(Xte, stats), stats

Read what it does not do. It never calls fit on Xte. The min, the mean, the median — every number the transform uses — comes from the training rows only, and the test rows are rescaled by those same numbers as if they'd arrived one at a time in production. The tempting shortcut is to scale the whole dataset once before splitting, because it's one line shorter and the code looks cleaner. Don't. That lets the test set's own min, mean, and spread seep into the transform your model trains against, and your held-out accuracy quietly inflates into a number you can't reproduce on data you haven't seen. Fit on train, transform everything — there's no exception to it.

Watch it work

This is the payoff. Below is a real k-nearest-neighbors vote on two Pima features — glucose across the bottom, diabetes-pedigree up the side — for one query point, the diamond. Its 15 nearest neighbors are ringed and wired to it by spokes; every point is colored by whether that patient had diabetes. The diamond is colored by whichever way its neighbors vote. The scene never moves a single data point. All that changes, frame by frame, is the ruler: we start in raw space, where distance is measured in native units and glucose — 95 times wider than pedigree — decides almost everything, and we slide to standardized space, where both features are measured in standard deviations and count equally.

Press play and watch the spokes. At the start the neighbors are whoever happens to share the query's glucose reading; the pedigree axis is effectively ignored, so the 15 nearest are a nearly vertical column of points. As the axes rescale, neighbors that were "close" only because their glucose matched get dropped, and points that are genuinely nearby once both features count get pulled in. The vote tally in the caption shifts as the membership churns.

The headline is in the caption. In raw space the 15 neighbors split 4 diabetes to 11 without, and the query is called no-diabetes. By the time the axes are standardized the very same query's 15 neighbors split 8 to 7, and the call flips to diabetes. Nothing about the patient changed. No model was retrained. We only stopped letting glucose's units shout down everything else, and a distance-based prediction reversed itself. That is the entire case for feature scaling, compressed into one point: on unscaled data, "nearest" was a lie told by the loudest column.

Reset and step through it one frame at a time to see the neighborhood turn over gradually rather than all at once — the membership changes at several points along the slide, not in a single jump, because different neighbors cross the boundary at different scales.

The full implementation

The whole file, no library — the three scalers and the no-leakage split, exactly as the checks and the animation used them:

"""Feature scaling, built from scratch.

Three ways to put columns on a common scale, each a fit/transform pair. `fit`
reads a statistic off the data (a min, a mean, a median); `transform` uses that
statistic to rescale. The split is the whole point of the chapter: you fit on
the training set and transform everything with those numbers, so the test set
never leaks its own statistics into the pipeline. Pure NumPy, no ML library
anywhere in this file.

Every function below appears in the chapter one step at a time (the
`# region:` markers are what the book's include directives pull in).
"""

import numpy as np
import pandas as pd


# region: minmax
def minmax_fit(X):
    """Learn the per-column min and max — the range of each feature.

    X is (N, D). Returns the two length-D vectors the transform needs. This is
    the only thing min-max "learns", and it learns it from whatever rows you
    hand it — so hand it the training set, never the whole dataset.
    """
    return {"min": X.min(axis=0), "max": X.max(axis=0)}


def minmax_transform(X, stats):
    """Squeeze every column into [0, 1] using a fitted range.

    Subtract the min so the smallest value maps to 0, divide by the span so the
    largest maps to 1. A column with no spread (max == min) would divide by
    zero, so we floor the denominator at 1 and leave that column at 0.
    """
    span = stats["max"] - stats["min"]
    span = np.where(span == 0, 1.0, span)
    return (X - stats["min"]) / span
# endregion


# region: standardize
def standardize_fit(X):
    """Learn the per-column mean and standard deviation.

    std uses ddof=0 (the population formula, dividing by N) to match
    scikit-learn's StandardScaler exactly. A zero-variance column gets its
    scale floored at 1 so the transform doesn't divide by zero.
    """
    std = X.std(axis=0)
    std = np.where(std == 0, 1.0, std)
    return {"mean": X.mean(axis=0), "std": std}


def standardize_transform(X, stats):
    """The z-score: subtract the mean, divide by the standard deviation.

    Every column comes out centered at 0 with unit spread, so a value of +2
    means "two standard deviations above this feature's mean" no matter which
    feature it is. That shared meaning is what makes columns comparable.
    """
    return (X - stats["mean"]) / stats["std"]
# endregion


# region: robust
def robust_fit(X):
    """Learn the per-column median and interquartile range (IQR).

    The median is the 50th percentile; the IQR is the 75th minus the 25th, the
    width of the middle half of the data. Neither one moves when a handful of
    points fly off to extreme values, which is what "robust" means here.
    np.percentile uses linear interpolation, matching sklearn's RobustScaler.
    """
    q25, q50, q75 = np.percentile(X, [25, 50, 75], axis=0)
    iqr = q75 - q25
    iqr = np.where(iqr == 0, 1.0, iqr)
    return {"median": q50, "iqr": iqr}


def robust_transform(X, stats):
    """Center on the median, scale by the IQR.

    Same shape as the z-score, but built from statistics that outliers can't
    drag around. The output isn't centered at exactly 0 mean or unit variance;
    it's centered at the median with the middle 50% spanning one unit.
    """
    return (X - stats["median"]) / stats["iqr"]
# endregion


# region: no_leak
def scale_train_test(Xtr, Xte, fit, transform):
    """Fit a scaler on TRAIN, apply it to both splits — no leakage.

    fit sees only Xtr, so the statistics (min/max, mean/std, median/IQR) are
    computed from training rows alone. The exact same fitted numbers rescale
    the test rows. If you instead fit on the full dataset, the test set's own
    distribution bleeds into the transform and your held-out score is a lie.
    """
    stats = fit(Xtr)
    return transform(Xtr, stats), transform(Xte, stats), stats
# endregion


def load_data(path="../data/diabetes.csv"):
    """The Pima diabetes set: 8 features on wildly different scales.

    Returns (X, y, feature_names): X is (N, 8) float, y is (N,) int 0/1
    (Outcome), names is the list of feature column names in order.
    """
    df = pd.read_csv(path)
    feature_names = [c for c in df.columns if c != "Outcome"]
    X = df[feature_names].to_numpy(float)
    y = df["Outcome"].to_numpy(int)
    return X, y, feature_names

The library version

You would not hand-roll these in production, and once you've seen the four-line versions you understand why nobody does. scikit-learn's preprocessing module ships all three as fit/transform objects with the same statistics we used. One helper picks the scaler, fits it on the training split, and transforms both:

def sklearn_scale(Xtr, Xte, kind):
    """Fit one of the three scalers on TRAIN, transform both splits.

    kind is 'minmax', 'standard', or 'robust'. Fitting on Xtr and transforming
    Xte with those fitted statistics is the no-leakage pattern the whole
    chapter is about — the scaler never sees the test rows while learning.
    """
    scaler = {"minmax": MinMaxScaler(),
              "standard": StandardScaler(),
              "robust": RobustScaler()}[kind]
    scaler.fit(Xtr)
    return scaler.transform(Xtr), scaler.transform(Xte)

MinMaxScaler, StandardScaler, and RobustScaler compute exactly the statistics our fit functions do — that's why the chapter's checkpoint asserts our transformed arrays match sklearn's to within 1e-9 on both splits, for all three scalers, before any chart is written. If a formula drifts, the run fails loudly instead of shipping a wrong picture. The real reason to use the library isn't a different answer; it's that a fitted scaler drops into a Pipeline, so the fit-on-train discipline is enforced by the framework instead of by you remembering. The downstream model we score everything against is a fixed k-NN classifier, held identical across runs so the only thing that ever changes is the scaling in front of it:

def knn_accuracy(Xtr, ytr, Xte, yte, k):
    """Test accuracy of a fixed k-NN classifier on whatever scaling it's given.

    The model is identical every time; only the preprocessing upstream changes.
    That isolation is the experiment: any change in this number is the scaling's
    doing, not the model's.
    """
    clf = KNeighborsClassifier(n_neighbors=k)
    clf.fit(Xtr, ytr)
    return float(clf.score(Xte, yte))

Scratch versus library

The usual face-off is a little different for a preprocessing chapter, because the scratch and library transforms are numerically identical by construction — the checkpoint proves it, so a bar chart of "our z-score versus sklearn's z-score" would be two bars of the same height. The comparison worth making is the one the whole chapter promised: what scaling does to a distance-based model. Same fixed k-NN classifier, same train/test split, four runs — raw features, then each scaler in front:

Read it honestly, because the numbers are honest. On raw features the k-NN classifier scores 0.745 test accuracy — fraction of the 192 held-out patients it labels correctly. Min-max lifts it to 0.755, and standardization to 0.771, a gain of 0.026 over unscaled, which on this split is five extra patients called right purely from changing the ruler. Robust scaling ties the raw features here at 0.745, a reminder that no scaler is automatically best — on this particular data the median/IQR version happened not to help the vote, and only measuring told me so. This isn't a dramatic ten-point swing; Pima's most predictive columns (glucose, insulin) are also its widest, so raw distance was accidentally leaning on the right features and unscaled k-NN was already halfway decent. The lift is modest and real, and standardization is the version that earned it.

The bigger lesson isn't the size of this particular bar. It's that the number moved at all from a transform that added zero information — no new features, no tuning, no retraining, just a change of units. On a dataset where the loud features were less lucky, that gap is the difference between a model that works and one that's secretly one-dimensional.

Takeaways

Feature scaling is the cheapest large lever in the whole pipeline, and it's invisible when it's missing — nothing errors, nothing warns, the model just quietly listens to whichever column has the biggest numbers. So carry the rule, not the vibe. If the model measures distance (k-NN, k-means, RBF SVMs, PCA), or learns by gradient descent (linear and logistic regression, neural nets), or penalizes coefficients (ridge, lasso), scale the features first — those methods assume comparable inputs and misbehave without them. If the model splits on thresholds (decision trees, random forests, gradient boosting), don't bother; any monotonic rescaling leaves every split unchanged and you've spent effort for nothing.

When you do scale, standardization is the default I reach for — unbounded, robust to nothing in particular but well-behaved for most columns, and it keeps outliers visible instead of crushing them into a box. Reach for min-max when a downstream component genuinely needs a bounded [0, 1] range, and for robust scaling when a column is riddled with outliers you can't clean, so the median and IQR carry the scale instead of a few extreme values. And whichever you pick, fit it on the training data alone and apply those frozen statistics to everything else. That one habit — no test statistics in the transform — is what separates a held-out score you can trust from a number that evaporates the moment real data arrives. Every distance-based and gradient-based method in the rest of this book assumes you've already done this. Now you have.