Beyond accuracy

A model that predicts "no diabetes" for everyone scores 65% on our test set — and learns nothing.

Accuracy is the first number everyone quotes and the first one that lies.

The scores, not the labels

A classifier hands you a probability, not a verdict. You choose where to cut.

The two classes overlap — and that overlap is every metric in this chapter.

\text{TP} = \#\{\hat{y}=1,\ y=1\}, \quad \text{FP} = \#\{\hat{y}=1,\ y=0\}, \quad \text{FN} = \#\{\hat{y}=0,\ y=1\}, \quad \text{TN} = \#\{\hat{y}=0,\ y=0\}

Precision $= \text{TP}/(\text{TP}+\text{FP})$ — of what we flagged, how much was real. Cost of a false alarm.
Recall $= \text{TP}/(\text{TP}+\text{FN})$ — of what was real, how much we caught. Cost of a miss.
Same numerator, different denominator. That swap is the whole game.
F1 = harmonic mean of the two. AUC = ranking quality, threshold-free.

All unitless, all in [0, 1].

Sweep the threshold 0 → 1. Confusion matrix, four metrics, and the ROC point move together.

Low threshold → recall 1, precision at the base rate. High threshold → the reverse.

No frame has every bar high. Where you stop is a decision about which mistake you'd rather make.

ROC bows above chance to AUC 0.83 — a random diabetic outranks a random healthy patient 83% of the time.

Our NumPy metrics land on sklearn's to the last decimal — asserted every run.

Accuracy 0.77 — but recall 0.61 means it misses 31 of 80 diabetics. Same model, opposite story.

Pick the metric that matches the cost of your error — before you train, not after.
Miss is expensive → recall. False alarm is expensive → precision. Ranking, no threshold yet → AUC.
Accuracy is honest only on balanced classes with equal error costs. Almost never the real case.
Always report the confusion matrix — every metric is a lossy summary of those four counts.
On imbalanced data, check accuracy against the majority baseline. Next chapter: when it lies outright.