Chapter 2 of 37 · basic

Turning data into features

What this chapter covers

A model never sees your data. It sees a matrix of numbers, and somebody had to turn the color column, the size column, the yes/no flag into that matrix before the algorithm ever ran. That translation is not bookkeeping. It's a modeling decision, and it's one of the few places where a single careless line quietly changes what the model is allowed to learn — no error, no warning, just a worse answer you can't explain later.

This chapter is about the translation. We start with the four kinds of column you meet in practice — a continuous number, a discrete count, a category with no order, a category with a real order, a yes/no — and build the encoders that turn each one into numbers by hand, in pure NumPy: one-hot for nominal categories, ordinal integers for genuinely ordered ones, and a single 0/1 column for the binary flag. Then scikit-learn's OneHotEncoder and OrdinalEncoder do the same job, and we confirm they land on the identical matrix. The point isn't any one encoder. It's the mistake that sits between two of them: giving a nominal category fake numeric order — red is 0, green is 1, blue is 2 — and watching a linear model believe every lie that ordering tells.

The data is a small synthetic product table, seeded so the encodings are legible and the effect is unmissable. And the payoff is one animation: the same color column, fit two ways, where a linear model's accuracy jumps from coin-flip to genuinely useful the moment we stop pretending colors have an order.

A bit of history

The idea that a category belongs in a regression at all goes back to the analysis of variance. R. A. Fisher, working at Rothamsted in the 1920s, needed a way to ask whether the fertilizer or the wheat variety changed the yield — and those are labels, not numbers. His answer, laid out in "Statistical Methods for Research Workers" in 1925, was to compare the variance between groups to the variance within them. What made it portable to the rest of statistics was the realization that ANOVA is just regression on indicator columns: you replace a k-level category with a set of 0/1 columns, one per level, and the regression coefficients become the group effects. Daniel Suits wrote the practitioner's version of this in 1957, in a paper titled plainly "Use of Dummy Variables in Regression Equations," and the word "dummy variable" is still what an econometrician calls the thing a machine-learning engineer calls one-hot.

The other name comes from hardware. In digital logic you often store a machine's state in a register where exactly one bit is high and all the rest are low — a one-hot register, so called because a single wire is "hot" at a time. It's wasteful of bits and wonderful for circuits: decoding the state is trivial because each state has its own dedicated line, with no arithmetic relating one to the next. That's precisely the property we want for a nominal feature. One column per category, exactly one of them on, and nothing about the encoding says any category is bigger, closer, or more than any other. Two traditions, a statistician counting harvest plots and an engineer wiring a state machine, arrived at the same trick for the same reason: some things are names, and names don't have arithmetic.

The intuition

Here's the whole problem in one sentence. A category is a name; a number has an order and a spacing; and the moment you write a name down as a number you have invented an order and a spacing the name never had.

Say the color of a product is red, green, or blue. Sort those alphabetically and hand out integers — blue is 0, green is 1, red is 2 — and you've told any model that reads them as numbers three things you never meant to say. That green sits between blue and red. That red is twice as far from blue as green is. That the step from blue to green is the same size as the step from green to red. None of that is true of colors. You made it up by alphabetizing. A linear model, which does nothing but add up weighted feature values, takes you completely at your word: it fits a single weight on that integer, and a single weight can only rank the categories in a straight line. If the truth is that red and blue behave alike and green is the odd one out, no straight line through 0, 1, 2 can say so.

One-hot encoding refuses to invent the order. Instead of one column with a fake number in it, you make one column per category, each holding a 0/1 flag for "is it this one." Now red, green, and blue sit on three separate axes, perpendicular and equidistant, and the model fits an independent weight for each. There's no "between," no "twice as far," nothing to be believed wrongly. The cost is width — a hundred cities become a hundred columns — but for a category with no real order it's the honest representation, and honesty is cheap here.

The math

A nominal variable takes one of $K$ values $c_1, \dots, c_K$ , and one-hot encoding maps a single value $x$ to a length- $K$ vector of indicators:

\operatorname{onehot}(x) = \big(\mathbb{1}[x = c_1],\ \mathbb{1}[x = c_2],\ \dots,\ \mathbb{1}[x = c_K]\big)

Here $\mathbb{1}[\cdot]$ is 1 when the condition holds and 0 otherwise, so the vector is all zeros except for a single 1 in the position of $x$ 's category. Every category becomes an axis of its own; no two are any closer than any other.

Contrast that with label (or ordinal) encoding, which assigns each category an integer $\operatorname{code}(x) \in \{0, 1, \dots, K-1\}$ and keeps a single column. A linear model's contribution from that column is one weight times that integer:

\eta = w \cdot \operatorname{code}(x) + b

Because $\eta$ is linear in the code, the predicted effect must march monotonically as the code increases — the model literally cannot make the middle category an outlier. Worse, any method that measures distance now reads the gap between two categories as

\big\lvert \operatorname{code}(c_i) - \operatorname{code}(c_j) \big\rvert

so categories coded 0 and 2 are treated as twice as far apart as 0 and 1. That imposed metric is fine when the codes reflect a real ordering with even spacing — a truly ordinal column like small, medium, large — and it is a fabrication when they don't. The math is the same either way; only your knowledge of the column decides whether it's a description or a lie.

What it's good at, what it isn't

One-hot's virtue is that it assumes nothing. It never sneaks an order into the model, it works for any nominal column, and the fitted weights read straight off as per-category effects. Its vice is width. A column with a thousand distinct values — user IDs, zip codes, product SKUs — becomes a thousand mostly-empty columns, which bloats memory, slows the fit, and hands a regularized linear model a thousand tiny knobs to overfit. That's the moment to reach past one-hot for a target encoder (replace each category with a smoothed average of the target) or a hashing encoder (map categories into a fixed number of columns and tolerate the collisions). High cardinality is one-hot's real limit, and it arrives faster than people expect.

Ordinal integers are the opposite bargain. They're compact — one column, always — and they carry exactly the information you want when the category has a genuine order with meaningful spacing. Small, medium, large; cold, warm, hot; the star rating on a review. Use integers there and you've helped the model, because the order is real and a linear term can use it. The failure is using that same compact encoding on a nominal column because it was one line shorter, at which point you've fabricated an order and the model believes it. The whole skill is telling the two cases apart, and it's not the code that tells you — it's whether the column has an order in the world.

The data

I built a small synthetic table on purpose, because a real dataset hides the mechanism under noise and I want the effect naked. Three hundred rows of a made-up product listing, one column of every type this chapter cares about:

price — a continuous number, dollars, drawn from a normal around 50.
reviews — a discrete count, a Poisson around 20.
color — a nominal category, red / green / blue, no order at all.
size — an ordinal category, S / M / L, a real order.
on_sale — a binary flag, yes / no.
converted — the target, 0 or 1, whether the visitor bought.

The generator is rigged so the signal in color is non-monotonic in any integer code: red and blue convert well, green does not. On the full table the conversion rate is 0.71 for blue, 0.07 for green, and 0.75 for red — the two ends high, the middle low. That's the XOR-shaped pattern that no single weight on an integer axis can rank correctly, which is the entire reason the encoding choice is going to matter here. The size effect, by contrast, is monotonic — larger sells more — so ordinal integers on size are honest. One column is a trap, one column is fine, and the difference is knowledge about the column, not the code.

Here's the mechanism itself. On the left, eight rows of the raw color column — one categorical value each. On the right, the same eight rows after one-hot encoding: three indicator columns, exactly one lit per row.

Read a row across. A red product becomes 0, 0, 1; a green one becomes 0, 1, 0. The single categorical cell has fanned out into a row of a matrix where exactly one entry is hot. Nothing is ordered, nothing is closer to anything else. This is the picture the history section was describing — the dummy variable and the one-hot register, the same trick drawn as a grid.

Build it, one function at a time

Each encoder is small, and the smallness is the point — there's nowhere for a subtle bug to hide. One-hot first. Fitting means learning the category vocabulary; transforming means lighting one column per row:

def one_hot_fit(col):
    """Learn the category vocabulary of a nominal column.

    col is a length-N array of category labels. We record the sorted set of
    distinct values; that fixed, ordered list is what pins each category to a
    specific output column so train and test line up. Sorting (not first-seen
    order) is what makes this match scikit-learn's OneHotEncoder exactly.
    """
    return sorted(set(col))


def one_hot_transform(col, categories):
    """Expand a nominal column into one 0/1 indicator column per category.

    Returns an (N, K) matrix where column k is 1 exactly on the rows whose value
    is categories[k], and 0 everywhere else. No category is closer to any other:
    every level sits on its own orthogonal axis, which is the whole point.
    """
    col = np.asarray(col)
    out = np.zeros((len(col), len(categories)), dtype=float)
    for k, cat in enumerate(categories):
        out[:, k] = (col == cat).astype(float)
    return out

I sort the categories rather than taking them in first-seen order for one boring, important reason: it's what makes train and test agree on which column is which, and it's what makes this match scikit-learn's encoder later so I can prove the two are identical. Ordinal encoding is the honest use of integers — you hand it the real order and it respects it:

def ordinal_encode(col, order):
    """Map a genuinely ordered column to integers that respect that order.

    `order` is the domain-knowledge ranking you supply, e.g. ["S", "M", "L"].
    Each value becomes its position in that list, so S->0, M->1, L->2. This is
    only honest when the column really is ordered: the integers assert that L is
    above M is above S, and that the step S->M is the same size as M->L.
    """
    rank = {cat: i for i, cat in enumerate(order)}
    return np.array([rank[v] for v in col], dtype=float)

Notice that ordinal_encode demands an order argument. You can't call it without stating, out loud, what the ranking is — S before M before L. That friction is a feature. It forces the one judgment that matters. Now the trap, the function that looks almost the same and means something completely different:

def label_encode(col):
    """Assign each category an integer by sorted order — no meaning attached.

    This is the mechanically identical twin of ordinal encoding, but with the
    order chosen alphabetically instead of by any real ranking. On a truly
    ordered column it happens to be fine; on a nominal column (color, city) it
    is the trap the chapter is about, because it invents an order the data never
    had: blue < green < red, with red twice as far from blue as green is.
    """
    categories = sorted(set(col))
    rank = {cat: i for i, cat in enumerate(categories)}
    return np.array([rank[v] for v in col], dtype=float), categories

label_encode is ordinal_encode with the order chosen for you, alphabetically, by nobody who knows the data. On a truly ordered column that's harmless luck. On a nominal column it's the whole bug: it invents blue < green < red and hands that fiction to the model as if it were measured. Same integers, same arithmetic — the only difference is whether the order it asserts exists in the world. Finally the binary flag, which needs exactly one column, not two:

def binary_encode(col, positive):
    """Collapse a two-valued column to a single 0/1 column.

    `positive` is the label that maps to 1 (everything else maps to 0). A binary
    variable needs exactly one column, not two — one indicator already carries
    all the information, and a second would be a perfect mirror of the first.
    """
    col = np.asarray(col)
    return (col == positive).astype(float)

A yes/no is already one bit of information; a single 0/1 column carries all of it, and a second column would just be its mirror image, redundant and, for a linear model, collinear. One column, done.

Watch it work

This is the payoff, and it's the cleanest demonstration in the book of a bad encoding doing measurable damage. We take the color column alone, fit a linear model to predict conversion from it, and show the fit two ways. The three hollow rings are the truth — each color's actual conversion rate, fixed, never moving: blue high, green near the floor, red high. The diamonds are the model's predictions, and the gray line through them is the model's response across the color axis.

Frame one is label encoding. Color has been flattened onto a single integer axis — blue at 0, green at 1, red at 2 — so the linear model's response has to be one straight-ish curve, and a straight curve through those three points can't dip in the middle. Watch what it does to green: green's true conversion rate is 0.07, down at the floor, but the model, forced to keep its response monotone, drags green's prediction up to 0.52, halfway up the plot, completely wrong. It isn't a tuning failure. The encoding made the right answer unreachable.

Press play and the fake order relaxes, frame by frame, into one-hot. As it does, green's diamond falls out of the line and drops toward its true rate of 0.07, landing at 0.10 once each color has its own axis. Blue and red settle onto their own true rates too. The gray line, which started as a nearly flat monotone drag, ends bent to match reality. And the number in the caption — accuracy, the fraction of the 300 rows this color-only model gets right — climbs from 0.52, barely better than guessing, to 0.79. Same column, same model, same data. All that changed is that we stopped telling the model green was between blue and red.

Reset and step through it one frame at a time. The prediction doesn't teleport; it slides, because we're blending two real fitted models — the label-encoded one and the one-hot one — and watching the linear model's forced compromise dissolve as the constraint lifts. The moment green crosses below 0.5 is the moment the accuracy jumps, because that's when the model finally starts calling green products "won't convert," which is what they are.

The full implementation

The whole encoders file, no library — the four transforms exactly as the checks and the animation used them:

"""Turning columns into features, built from scratch.

A model only ever sees a matrix of numbers. Everything upstream of that matrix
is a decision about how a raw column — a color, a size, a yes/no flag — becomes
those numbers, and the decision is not free: it tells the model what kind of
thing the column is. This file builds the four encoders you reach for by hand,
in pure NumPy (+ pandas for loading), so the mechanics are in front of you:

  one_hot   nominal categories  -> one indicator column each
  ordinal   truly ordered levels -> integers that respect the order
  label     categories           -> sorted integers (the trap on nominal data)
  binary    a two-valued column  -> a single 0/1 column

Every function below appears in the chapter one step at a time (the
`# region:` markers are what the book's include directives pull in).
"""

import numpy as np
import pandas as pd


# region: onehot
def one_hot_fit(col):
    """Learn the category vocabulary of a nominal column.

    col is a length-N array of category labels. We record the sorted set of
    distinct values; that fixed, ordered list is what pins each category to a
    specific output column so train and test line up. Sorting (not first-seen
    order) is what makes this match scikit-learn's OneHotEncoder exactly.
    """
    return sorted(set(col))


def one_hot_transform(col, categories):
    """Expand a nominal column into one 0/1 indicator column per category.

    Returns an (N, K) matrix where column k is 1 exactly on the rows whose value
    is categories[k], and 0 everywhere else. No category is closer to any other:
    every level sits on its own orthogonal axis, which is the whole point.
    """
    col = np.asarray(col)
    out = np.zeros((len(col), len(categories)), dtype=float)
    for k, cat in enumerate(categories):
        out[:, k] = (col == cat).astype(float)
    return out
# endregion


# region: ordinal
def ordinal_encode(col, order):
    """Map a genuinely ordered column to integers that respect that order.

    `order` is the domain-knowledge ranking you supply, e.g. ["S", "M", "L"].
    Each value becomes its position in that list, so S->0, M->1, L->2. This is
    only honest when the column really is ordered: the integers assert that L is
    above M is above S, and that the step S->M is the same size as M->L.
    """
    rank = {cat: i for i, cat in enumerate(order)}
    return np.array([rank[v] for v in col], dtype=float)
# endregion


# region: label
def label_encode(col):
    """Assign each category an integer by sorted order — no meaning attached.

    This is the mechanically identical twin of ordinal encoding, but with the
    order chosen alphabetically instead of by any real ranking. On a truly
    ordered column it happens to be fine; on a nominal column (color, city) it
    is the trap the chapter is about, because it invents an order the data never
    had: blue < green < red, with red twice as far from blue as green is.
    """
    categories = sorted(set(col))
    rank = {cat: i for i, cat in enumerate(categories)}
    return np.array([rank[v] for v in col], dtype=float), categories
# endregion


# region: binary
def binary_encode(col, positive):
    """Collapse a two-valued column to a single 0/1 column.

    `positive` is the label that maps to 1 (everything else maps to 0). A binary
    variable needs exactly one column, not two — one indicator already carries
    all the information, and a second would be a perfect mirror of the first.
    """
    col = np.asarray(col)
    return (col == positive).astype(float)
# endregion


def load_data(path="../data/products.csv"):
    """The synthetic product table: one column of every type we care about.

    Returns the DataFrame as-is. Columns:
      price     continuous numeric   (float)
      reviews   discrete numeric     (int count)
      color     nominal category     (red / green / blue)
      size      ordinal category     (S < M < L)
      on_sale   binary               (yes / no)
      converted target               (0 / 1 — did the visitor buy)
    """
    return pd.read_csv(path)

The library version

Nobody hand-rolls these in production, and once you've seen the from-scratch versions you know why the library ones are safe to trust: they do the identical thing. scikit-learn's preprocessing module ships OneHotEncoder and OrdinalEncoder as fit/transform objects with the same category logic we used — sorted vocabulary, one column per level for one-hot, an explicit order for ordinal:

def sklearn_one_hot(col):
    """scikit-learn's OneHotEncoder on one nominal column.

    sparse_output=False gives a dense (N, K) array; categories default to the
    sorted unique values, so the column order matches our one_hot exactly.
    """
    enc = OneHotEncoder(sparse_output=False, categories="auto")
    return enc.fit_transform(np.asarray(col).reshape(-1, 1)), list(enc.categories_[0])


def sklearn_ordinal(col, order):
    """scikit-learn's OrdinalEncoder with an explicit category order.

    Passing categories=[order] pins S->0, M->1, L->2 the way our ordinal_encode
    does. With the default (sorted) order this same object is 'label' encoding.
    """
    enc = OrdinalEncoder(categories=[list(order)])
    return enc.fit_transform(np.asarray(col).reshape(-1, 1)).ravel()

The chapter's checkpoint asserts our scratch one-hot matches OneHotEncoder's output cell for cell — same column order, same 0/1 values, every row summing to exactly one — and that our ordinal encoding matches OrdinalEncoder's when given the same S/M/L order, before any chart is written. If an encoder drifts, the run fails loudly instead of shipping a wrong picture. The two downstream models we score everything against are held fixed, so the only thing that ever changes between runs is how color was encoded in front of them — a linear model that adds up weighted columns, and a tree that splits on thresholds:

def logistic_fit_score(Xtr, ytr, Xte, yte):
    """Test accuracy of a fixed logistic-regression classifier.

    A linear model: it can only add up a weighted sum of the feature columns.
    That linearity is exactly what makes it believe an ordinal integer's fake
    distances, which is what we want to expose.
    """
    clf = LogisticRegression(max_iter=2000, random_state=0)
    clf.fit(Xtr, ytr)
    return float(clf.score(Xte, yte))


def tree_fit_score(Xtr, ytr, Xte, yte):
    """Test accuracy of a fixed decision-tree classifier.

    A tree splits one column at a time on thresholds, so it can carve a single
    integer axis into per-category bands (code <= 0.5, code <= 1.5, ...). It is
    far less fooled by a bad encoding than the linear model is.
    """
    clf = DecisionTreeClassifier(max_depth=4, random_state=0)
    clf.fit(Xtr, ytr)
    return float(clf.score(Xte, yte))

That pairing is deliberate. The linear model is the one the fake order fools; the tree is the control that shows the damage is specific to models that do arithmetic on their inputs.

Scratch versus library

The usual face-off is a little different for an encoding chapter, because the scratch and library encoders are identical by construction — the checkpoint proves it, so charting "our one-hot versus sklearn's one-hot" would be two bars of the same height. The comparison worth making is the one the chapter promised: what the encoding choice does to a real model's held-out accuracy. Same synthetic table, one 225-row train / 75-row test split, four runs — the linear model and the tree, each fed color as a label integer (wrong) and as one-hot columns (right):

Read the two pairs. For the linear model, label-encoding color scores 0.573 test accuracy — the fraction of the 75 held-out visitors it labels correctly — and switching to one-hot lifts it to 0.760, a gain of 0.187, fourteen extra visitors called right out of seventy-five, from nothing but a change of encoding. No new features, no tuning, no retraining the model differently. We just stopped lying to it about color, and a coin-flip classifier became a useful one. That 0.187 is the entire chapter, printed as a bar.

Now look at the tree. Label-encoded, it scores 0.747; one-hot, 0.733 — a swing of 0.013, statistical noise. The tree barely cares, and it's worth understanding why, because it's the same reason it didn't care about scaling last chapter. A tree splits one column at a time on thresholds, so even handed the fake integer axis it can cut it into per-category bands — code below 0.5 is blue, below 1.5 is green, the rest is red — and recover the categories the linear model couldn't. The lesson isn't that encoding never matters; it's that it matters for models that do arithmetic on their inputs and matters far less for models that ask yes/no questions about them. If you only ever ran trees you could get sloppy here. The day you put a linear or a distance-based model behind that same label integer, the sloppiness costs you 0.187.

Takeaways

Match the encoding to what the variable actually is, not to what's least typing. If a category has no order — color, city, product line, browser — it's nominal, and it wants one-hot: one column per level, no arithmetic relating them, nothing for a linear model to misread. If a category has a real, evenly-spaced order — small/medium/large, a survey's disagree-to-agree scale, a severity grade — then ordinal integers are not just acceptable, they're the right call, because the order is information and integers hand it to the model for free. The failure mode is exactly one substitution: reaching for the compact integer encoding on a nominal column because it was one line shorter. That's the "red is 0, green is 1, blue is 2" trap, and this chapter measured what it costs — 0.187 of accuracy on a linear model, the difference between a model that works and one that doesn't.

Two caveats I carry into every project. First, one-hot has a ceiling: high cardinality. A few dozen categories is fine; a few thousand user IDs is a memory-eating, overfitting mess, and that's where target encoding or a hashing encoder earns its keep. Know the switch-over point before you hit it. Second, the model behind the encoding decides how much any of this matters — linear and distance-based and gradient-descent models take your fabricated order at face value and suffer, while trees and tree ensembles mostly shrug it off. So the question is never just "how do I encode this column," it's "how do I encode this column for the model I'm about to feed it." Get the type right, respect real orders and refuse to invent fake ones, and watch the cardinality. That's the whole discipline, and it happens before the algorithm you actually came for ever runs.