When accuracy lies

A model that predicts the majority class for all 600 test rows scores 0.955 accuracy — and finds none of the cases it exists to find.

At 4.5% prevalence, accuracy is not a measurement. It is the imbalance ratio wearing a lab coat.

The accuracy paradox

\mathrm{acc}_{\text{majority}} = \frac{\max_c n_c}{n} = \max_c \Pr(y = c)

The do-nothing model's accuracy = the majority fraction.
On a 95/5 split it starts at 0.95. A real model has to beat that floor, not zero.
"Accuracy paradox": throwing away the classifier and always guessing the majority can raise accuracy.

107 minority points in 2000 (~5%), sitting inside the edge of the majority cloud.

The cheapest way to be right is to color the plane cyan. That is what a naive model drifts toward.

Random OVERSAMPLING — copy minority rows until the classes match. Keeps data, risks memorizing.
Random UNDERSAMPLING — drop majority rows to match. Fast, throws real data away.
CLASS WEIGHTS — reweight the loss, no data touched.

\mathcal{L}_{w} = -\frac{1}{n}\sum_{i=1}^{n} w_{y_i}\big[\,y_i \log \hat{p}_i + (1 - y_i)\log(1 - \hat{p}_i)\,\big]

w_c = \frac{n}{k\,n_c}

On our train split: minority weight 8.75, majority 0.53. Resample the TRAIN set only.

Naive: accuracy 0.96, recall 0.19 — catches 5 of 27. Then each fix swings the boundary over the cluster.

Recall goes 0.19 → 1.0. Accuracy slips 0.96 → 0.82 (four points above a 0.955 floor). AUC does not move.

Accuracy barely moves; recall transforms; precision pays the bill; AUC stays flat.

Flat AUC ≈ 0.94 = same ranker, different operating point. The fixes move the line, not the model.

Never report bare accuracy on imbalanced data — compute the majority baseline first.
Pick the metric that matches the rare-class cost: miss expensive → recall; false alarm expensive → precision.
Resample the TRAIN set only, never the test set. Seed the resampling.
Class weights are the cheapest fix — one argument, all the data.
The AUC won't move? The model already ranks fine. Calibrate the threshold to the business cost, not to 0.50.