Missing values and messy data

Find and fix the problems in real data before any model runs.

A missing value doesn't have to be blank to be missing. Sometimes it's a zero hiding in plain sight.

The Pima trap

A glucose, blood pressure, skin fold, insulin, or BMI of 0 is physically impossible.
So a 0 in those columns isn't a measurement — it's a missing value in disguise.
Load the raw table and your model believes half its patients produce no insulin.
That's not a fact about diabetes. It's a fact about data entry.

652 disguised-missing cells in the "clean" Pima benchmark. Insulin is 48.7% fake.

Fill with the mean or the median of the observed values:

\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i \qquad \operatorname{median}(x) = x_{((n+1)/2)}

Two outlier rules that disagree on purpose:

\text{z-score: flag } |x_i - \bar{x}| / \sigma > 3

\text{IQR: flag } x_i < Q_1 - 1.5\,\text{IQR} \;\text{ or }\; x_i > Q_3 + 1.5\,\text{IQR}

The z-score's mean and std are inflated by the very outliers it hunts, so it under-flags a skewed column. IQR doesn't.

Real NaNs, plus zeros in the columns where zero is physically impossible.

Insulin is 48.7% fake zeros and right-skewed. Mean fill parks at 156 in the empty tail; median fill lands at 125 in the bulk.

Fixed classifier, one split. Test accuracy — fraction of 192 held-out patients called right (unitless, 0–1).

Raw zeros 0.781, naive impute drops to 0.750 — the missingness was informative. Median + a was-missing flag recovers it: 0.771.

Look at your data before you model it — column by column, at the zeros that can't be true.
Disguised missingness is everywhere zero is a legal value for a column that can't be zero. Domain knowledge finds it, not a null check.
Median-impute skewed columns; mean piles guesses out in the empty tail.
Add a was-missing indicator when the absence carries signal — it saved the cleaned model here.
Fit the fill on train, apply everywhere. Garbage in, garbage out beats any algorithm choice.