Turning data into features

A model never sees your data — it sees a matrix of numbers.

How a column becomes those numbers is a modeling decision, not bookkeeping.

The four kinds of column

• Numeric — continuous (price) or discrete counts (reviews). Already numbers.
• Nominal — a category with no order (color: red / green / blue).
• Ordinal — a category with a real order (size: S < M < L).
• Binary — a two-valued flag (on_sale: yes / no).

The type of the column decides how you're allowed to encode it.

The trap

Write a name down as a number and you invent an order it never had.

Label-encode color as blue = 0, green = 1, red = 2 and you've told the model:

• green sits between blue and red,
• red is twice as far from blue as green is,
• the steps are evenly spaced.

None of that is true of colors. A linear model believes all of it.

One-hot refuses to invent the order

One column per category, exactly one lit per row — three orthogonal axes, nothing closer to anything.

Why the integer lies

A linear model's contribution from a label integer is one weight times the code:

Its response must be monotone in the code — it cannot make the middle category an outlier. And any distance reads

so 0 and 2 are twice as far apart as 0 and 1. Fine for real orders, a fabrication otherwise.

Watch the fake order dissolve

Color alone, fit two ways. Rings = true rates; diamonds = the model. Green's true rate is 0.07.

Label-encoded, green is dragged to 0.52. One-hot, it drops to 0.10. Accuracy 0.52 → 0.79.

What it costs the model

Same table, one split, four runs. Test accuracy — fraction of held-out visitors called right (unitless, 0–1).

Linear model: 0.573 → 0.760, a 0.187 lift. Tree: 0.747 → 0.733, barely a flinch.

Takeaways

• Match the encoding to the variable's true nature — one-hot for nominal, ordinal integers only for truly ordered.
• The "red = 0, green = 1, blue = 2" trap fabricates an order; a linear model believes every bit of it.
• High-cardinality categories break one-hot — reach for target or hashing encoders.
• Trees shrug off bad encodings; linear and distance-based models pay full price.