Judging a classifier: beyond accuracy

What this chapter covers

Accuracy is the first number everyone reaches for and the first number that lies to them. A model that predicts "no diabetes" for all 231 patients in our test set scores 65% accuracy without learning anything, because 65% of those patients don't have diabetes. Report that 65% with a straight face and you've technically told the truth and completely misled the room.

So this chapter is about all the other numbers — the ones that tell you not just how often the model is right, but which way it's wrong. We build the whole family by hand in NumPy: the confusion matrix, precision, recall, specificity, F1, and the ROC curve with its area underneath. Then we let scikit-learn compute the same things and check that every digit agrees. This isn't an algorithm chapter. Nothing here trains a model. A logistic regression already produced the scores; our job is to judge them, and to know what each number measures before we trust it.

The centerpiece is an animation. We slide the decision threshold from 0 to 1 and watch the confusion matrix, the four metrics, and a point on the ROC curve move together, so you can see the precision-recall trade-off happen instead of reading about it. It's the direct sequel to the confusion-matrix sweep in the threshold chapter — same idea, more instruments on the dashboard.

A bit of history

The ROC curve is a piece of World War II hardware that wandered into statistics. ROC stands for receiver operating characteristic, and the receiver in question was a radar operator staring at a scope, deciding whether a blip was an incoming bomber or a flock of geese. Turn the sensitivity up and you catch every plane but also jump at every goose; turn it down and the false alarms stop but you miss real targets. Engineers plotted that trade-off as a curve — hits against false alarms as the detection threshold moved — and the curve outlived the war. Signal detection theory carried it into psychology in the 1950s, radiology adopted it to grade diagnostic tests, and machine learning inherited it wholesale. When you compute an AUC today you're using a tool built to tune radar.

Precision and recall come from a different world: information retrieval. In the 1950s and 60s, people building the first document-search systems needed to say how good a search was, and "accuracy" is meaningless when the right answer is 12 documents out of a million. Cyril Cleverdon's Cranfield experiments in the late 1950s pinned down the pair we still use — recall for how many of the relevant documents you found, precision for how many of the ones you returned were actually relevant. Every search engine, every recommender, every retrieval system since is graded on some descendant of those two numbers. F1, the harmonic mean that folds them into one, showed up later as a convenience for when you want a single knob to rank systems by.

The intuition

A classifier like logistic regression doesn't hand you a label. It hands you a score — an estimated probability that the patient is diabetic — and you decide where to cut. Everything about judging the model starts from one picture: the scores it gave the patients who really were diabetic, against the scores it gave the ones who weren't.

The two humps lean apart — the model does push the real diabetics toward higher scores — but they overlap in the middle, and that overlap is the whole story. Wherever you drop the threshold line, some blue (healthy) sits to its right and some purple (diabetic) sits to its left. Those are your two kinds of mistake, and no single line makes both disappear. Slide the line left and you scoop up almost every diabetic, along with a crowd of healthy people. Slide it right and the healthy crowd clears out, but diabetics start slipping under it. Accuracy collapses that entire trade-off into one number and throws the shape away. The metrics in this chapter are ways to keep the shape.

The math

Every metric is built from four counts. Fix a threshold, label each patient, and sort each one into one of four bins by comparing the prediction to the truth. Call the positive class 1 (diabetic):

TP is a caught case, TN a correctly cleared one; FP is a false alarm, FN a missed case. Those four integers arranged in a 2×2 grid are the confusion matrix, and every number below is a ratio of them.

Accuracy is the diagonal over the total — the fraction of all calls that were right:

Precision asks: of everyone we flagged, how many really were positive. Recall asks: of everyone who really was positive, how many did we catch. They divide the same TP by different denominators, and that difference is everything:

Specificity is recall's mirror on the other class — of everyone who really was negative, how many we cleared:

F1 folds precision and recall into one score using the harmonic mean, which sits near the smaller of the two so you can't fake it by acing one and tanking the other:

Those are all point metrics — they judge one threshold. The ROC curve judges the model at every threshold at once. Sweep the cutoff from high to low and plot the true positive rate against the false positive rate:

The area under that curve is the AUC, which we get by trapezoidal integration over the curve's points:

One unit note that this whole chapter turns on: accuracy, precision, recall, specificity, and F1 are unitless fractions in [0, 1] — each is a count divided by a count. AUC is also unitless in [0, 1], but it isn't a fraction of the data; it's a probability, the chance that a random positive gets a higher score than a random negative. Half of using these metrics well is remembering which count sits in each denominator.

What each metric measures, and what it hides

Accuracy is the honest choice exactly when the classes are balanced and the two mistakes cost the same — and almost nowhere else. On our test set 65% of patients are healthy, so the do-nothing model that clears everyone already scores 65%. Any metric that a brick can pass is a bad yardstick for a model. Accuracy also can't tell a cautious model from a trigger-happy one: two classifiers with the same accuracy can have wildly different false-alarm rates, and accuracy shows you none of it.

Precision and recall split that blind spot open, one per kind of mistake. Precision is what you care about when a false positive is expensive — a spam filter that dumps real mail, a fraud flag that freezes a real customer. Recall is what you care about when a miss is the disaster — a cancer screen, a diabetes test, anything where the cost of overlooking a real case dwarfs the cost of a second look. You can trade one for the other freely by moving the threshold, so neither means anything without the other pinned down; quoting a precision without its recall is a magic trick, not a measurement. F1 ties them together when you genuinely have no reason to favor either, but it also buries the trade-off it averaged over, so I treat it as a ranking convenience, not a target. AUC steps out of the threshold game entirely and grades the ranking: does the model tend to score positives above negatives, wherever you'd cut? It's the right summary when you haven't chosen an operating point yet, and a misleading one on heavily imbalanced data, where a fat pile of easy true negatives can prop the number up. That's the failure the imbalanced-classes chapter is built around.

The data

The scores come from a logistic regression fit on the Pima Indians Diabetes set — the same 768 patients and eight measurements as the threshold chapter, split 70/30 into 537 training and 231 test patients. Of the 231 held-out patients, 80 were diabetic, so the positive class is 34.6% of the test set: imbalanced enough that accuracy and the rest pull visibly apart, which is the whole reason to use this dataset here.

The model itself barely matters for this chapter — we could judge any classifier's scores the same way. What matters is that predict_proba gives a real probability per patient, committed to data/scores.json alongside the true label, and every number on this page traces back to those 231 pairs. The score distribution you saw above is a smoothed picture of exactly that file.

Build it, one function at a time

Everything starts with the four counts. Threshold the scores into 0/1 predictions, then tally each patient into one of four bins:

def confusion_counts(y_true, y_pred):
    """The four counts a binary confusion matrix is made of.

    Compares hard 0/1 predictions against 0/1 truth and returns
    (tp, fp, fn, tn):

      TP  true positive   predicted 1, truly 1   (caught a real case)
      FP  false positive  predicted 1, truly 0   (a false alarm)
      FN  false negative  predicted 0, truly 1   (a missed case)
      TN  true negative   predicted 0, truly 0   (correctly cleared)

    Every metric below is some ratio of these four integers.
    """
    y_true = np.asarray(y_true).astype(int)
    y_pred = np.asarray(y_pred).astype(int)
    tp = int(np.sum((y_pred == 1) & (y_true == 1)))
    fp = int(np.sum((y_pred == 1) & (y_true == 0)))
    fn = int(np.sum((y_pred == 0) & (y_true == 1)))
    tn = int(np.sum((y_pred == 0) & (y_true == 0)))
    return tp, fp, fn, tn

That thresholding step is its own one-liner, because we're about to call it at fifty different cutoffs and I want the decision rule in exactly one place:

def predict_at(y_score, threshold):
    """Turn continuous scores into 0/1 labels at a decision threshold.

    Predict the positive class when the score is at or above the threshold.
    Everything downstream is a function of where you put this line.
    """
    return (np.asarray(y_score) >= threshold).astype(int)

With the four counts in hand, every point metric is a short division. Accuracy is the diagonal over the total:

def accuracy(tp, fp, fn, tn):
    """Fraction of all predictions that were correct: the diagonal / total.

    Unitless, in [0, 1]. Reads well, lies on imbalanced data: predict the
    majority class for everyone and this number can still look high.
    """
    total = tp + fp + fn + tn
    return (tp + tn) / total if total else 0.0

Precision divides the true positives by everything we flagged; recall divides the same true positives by everything that was actually positive. Same numerator, different denominator — that one swap is the entire difference between the two most confused numbers in machine learning:

def precision(tp, fp):
    """Of everything we FLAGGED as positive, what fraction really was.

    precision = TP / (TP + FP). Unitless, in [0, 1]. The cost of a false
    alarm lives here — low precision means you're crying wolf. Undefined
    when nothing is flagged; we return 0 to match sklearn's convention.
    """
    return tp / (tp + fp) if (tp + fp) else 0.0

def recall(tp, fn):
    """Of everything that truly WAS positive, what fraction we caught.

    recall = TP / (TP + FN), a.k.a. sensitivity or the true positive rate.
    Unitless, in [0, 1]. The cost of a miss lives here — low recall means
    real cases slipped through. Undefined when there are no positives.
    """
    return tp / (tp + fn) if (tp + fn) else 0.0

Specificity is recall read on the negative class, and it's the piece the ROC curve needs — its false positive rate is just one minus this:

def specificity(tn, fp):
    """Of everything that truly was NEGATIVE, what fraction we cleared.

    specificity = TN / (TN + FP), the true negative rate. Unitless, in
    [0, 1]. Recall's mirror image on the other class; 1 - specificity is
    the false positive rate, the x-axis of the ROC curve.
    """
    return tn / (tn + fp) if (tn + fp) else 0.0

F1 takes the harmonic mean of precision and recall. The harmonic mean, not the plain average, on purpose: it drags toward whichever of the two is smaller, so a model can't buy a good F1 by maxing one and abandoning the other:

def f1(precision_val, recall_val):
    """The harmonic mean of precision and recall — one number for both.

    f1 = 2 * P * R / (P + R). Unitless, in [0, 1]. Harmonic, not
    arithmetic, on purpose: it stays near the smaller of the two, so you
    can't win F1 by acing precision while recall collapses. Zero when
    either is zero.
    """
    denom = precision_val + recall_val
    return 2 * precision_val * recall_val / denom if denom else 0.0

The animation needs a full snapshot at each threshold, so one wrapper computes everything at once and hands back a dict — including the TPR and FPR the ROC point rides on:

def metrics_at(y_true, y_score, threshold):
    """Every scalar metric at one decision threshold, in one call.

    Threshold the scores, count the four cells, and derive the ratios.
    Returns a dict so the animation can read a full metric snapshot per
    frame without recomputing anything.
    """
    y_pred = predict_at(y_score, threshold)
    tp, fp, fn, tn = confusion_counts(y_true, y_pred)
    p = precision(tp, fp)
    r = recall(tp, fn)
    return {
        "threshold": float(threshold),
        "tp": tp, "fp": fp, "fn": fn, "tn": tn,
        "accuracy": accuracy(tp, fp, fn, tn),
        "precision": p,
        "recall": r,
        "specificity": specificity(tn, fp),
        "f1": f1(p, r),
        "tpr": r,                       # recall == true positive rate
        "fpr": 1.0 - specificity(tn, fp),
    }

Now the curves. The ROC sweep walks every distinct score from high to low. The only subtlety is grouping tied scores into a single step, which is what makes the trapezoidal area come out to the exact AUC rather than an approximation:

def roc_curve(y_true, y_score):
    """Sweep every threshold and trace (FPR, TPR) — the ROC curve.

    The thresholds that matter are the distinct score values: between two
    adjacent scores nothing changes. Walk them from high to low, so the
    curve starts at (0, 0) (threshold above every score, nothing flagged)
    and ends at (1, 1) (threshold below every score, everything flagged).
    Tied scores are grouped into a single step, which is what makes the
    trapezoidal area equal the true AUC. Returns (fpr, tpr, thresholds).
    """
    y_true = np.asarray(y_true).astype(int)
    y_score = np.asarray(y_score, dtype=float)
    P = int(np.sum(y_true == 1))
    N = int(np.sum(y_true == 0))
    thresholds = np.sort(np.unique(y_score))[::-1]   # high -> low
    fpr = [0.0]
    tpr = [0.0]
    thr = [np.inf]
    for t in thresholds:
        y_pred = (y_score >= t).astype(int)
        tp = int(np.sum((y_pred == 1) & (y_true == 1)))
        fp = int(np.sum((y_pred == 1) & (y_true == 0)))
        tpr.append(tp / P if P else 0.0)
        fpr.append(fp / N if N else 0.0)
        thr.append(float(t))
    return np.array(fpr), np.array(tpr), np.array(thr)

The area itself is one trapezoid rule over those points:

def auc(fpr, tpr):
    """Area under the ROC curve by the trapezoidal rule.

    Integrate TPR with respect to FPR. Unitless, in [0, 1], and it has a
    clean meaning: the probability that a random positive is scored above a
    random negative. 0.5 is a coin flip, 1.0 is a perfect ranking. Because
    the curve is monotone in FPR, trapezoids give the exact area.
    """
    return float(np.trapezoid(tpr, fpr))

And the precision-recall curve is the same sweep read on different axes — precision against recall instead of TPR against FPR:

def pr_curve(y_true, y_score):
    """The precision-recall curve: precision vs recall as the threshold sweeps.

    Same threshold sweep as the ROC, read on different axes. On imbalanced
    data this curve is the honest one — it ignores the huge true-negative
    count that flatters the ROC. Returns (recall_points, precision_points).
    """
    y_true = np.asarray(y_true).astype(int)
    y_score = np.asarray(y_score, dtype=float)
    thresholds = np.sort(np.unique(y_score))[::-1]
    recalls, precisions = [], []
    for t in thresholds:
        y_pred = (y_score >= t).astype(int)
        tp, fp, fn, tn = confusion_counts(y_true, y_pred)
        recalls.append(recall(tp, fn))
        precisions.append(precision(tp, fp))
    return np.array(recalls), np.array(precisions)

Watch it work

Here's the whole point of the chapter in one animation. We sweep the decision threshold from 0 to 1 in steps of 0.02 across the 231 real test patients. Each frame is one threshold, computed live: the confusion matrix on the left, the four metrics as bars in the middle, and the operating point tracing the ROC curve on the bottom. The caption names the threshold and reads out each metric with its meaning.

Press play and watch the trade-off you can't escape. Start at the left, with the threshold near 0: the model flags everyone, so recall is a perfect 1.0 — no diabetic is missed — but precision sinks to the base rate, because most of the people you flagged are healthy. The ROC point sits up at the top-right corner, (1, 1), all hits and all false alarms. Now drag the threshold up. False alarms drain out of the bottom-left cell, precision climbs, and the point walks down the curve toward the origin — but recall falls the whole way as diabetics start landing in the missed-case cell. Accuracy humps in the middle and then sags. There is no frame where every bar is high at once. Where you stop the sweep is a decision about which mistake you'd rather make, and that's the confusion-matrix trade-off from the threshold chapter, now with the metrics that name it wired to the same dial.

The curve the point rides on is worth seeing whole. Here's the full ROC with its area shaded — the AUC — and next to it the precision-recall curve for the same model on the same scores.

The dashed diagonal is chance — the ROC of a coin flip, AUC 0.5. Our curve bows up above it to an AUC of 0.83, meaning that if you pick one diabetic and one healthy patient at random, the model gives the diabetic the higher score about 83% of the time. That's a ranking quality, independent of any threshold, which is exactly why AUC is the number to quote when you haven't chosen an operating point yet.

The precision-recall curve tells the same story from the working end. Its dashed line is the base rate, 0.346 — what precision a random guesser would hit — and a useful model lives above it. Read it right to left: push recall toward 1 and precision decays toward the base rate; pull back to high precision and you give up recall. On imbalanced problems this curve is the more honest of the two, because it never lets the giant pile of easy true negatives flatter the score the way the ROC's false-positive-rate axis quietly can.

The full implementation

The whole metrics library, no dependencies past NumPy, top to bottom. This is the file the animation and both curves actually ran:

"""Classification metrics, built from scratch.

Given true labels and a classifier's predicted scores, every number you'd
ever use to judge it: the confusion matrix, accuracy, precision, recall,
specificity, F1, the ROC curve and its AUC, and the precision-recall curve.
Pure NumPy — no metrics library anywhere in this file.

None of this trains a model. A model already produced the scores; this file
is only about scoring the scoring. Every function below appears in the
chapter one step at a time (the `# region:` markers are what the book's
include directives pull in).

Convention throughout: label 1 is the positive class (the thing we're trying
to catch — a diabetic patient), label 0 is the negative class. A "score" is
the model's estimated P(y = 1); a decision needs a threshold applied to it.
"""

import numpy as np


# region: confusion
def confusion_counts(y_true, y_pred):
    """The four counts a binary confusion matrix is made of.

    Compares hard 0/1 predictions against 0/1 truth and returns
    (tp, fp, fn, tn):

      TP  true positive   predicted 1, truly 1   (caught a real case)
      FP  false positive  predicted 1, truly 0   (a false alarm)
      FN  false negative  predicted 0, truly 1   (a missed case)
      TN  true negative   predicted 0, truly 0   (correctly cleared)

    Every metric below is some ratio of these four integers.
    """
    y_true = np.asarray(y_true).astype(int)
    y_pred = np.asarray(y_pred).astype(int)
    tp = int(np.sum((y_pred == 1) & (y_true == 1)))
    fp = int(np.sum((y_pred == 1) & (y_true == 0)))
    fn = int(np.sum((y_pred == 0) & (y_true == 1)))
    tn = int(np.sum((y_pred == 0) & (y_true == 0)))
    return tp, fp, fn, tn
# endregion


# region: predict_at
def predict_at(y_score, threshold):
    """Turn continuous scores into 0/1 labels at a decision threshold.

    Predict the positive class when the score is at or above the threshold.
    Everything downstream is a function of where you put this line.
    """
    return (np.asarray(y_score) >= threshold).astype(int)
# endregion


# region: accuracy
def accuracy(tp, fp, fn, tn):
    """Fraction of all predictions that were correct: the diagonal / total.

    Unitless, in [0, 1]. Reads well, lies on imbalanced data: predict the
    majority class for everyone and this number can still look high.
    """
    total = tp + fp + fn + tn
    return (tp + tn) / total if total else 0.0
# endregion


# region: precision
def precision(tp, fp):
    """Of everything we FLAGGED as positive, what fraction really was.

    precision = TP / (TP + FP). Unitless, in [0, 1]. The cost of a false
    alarm lives here — low precision means you're crying wolf. Undefined
    when nothing is flagged; we return 0 to match sklearn's convention.
    """
    return tp / (tp + fp) if (tp + fp) else 0.0
# endregion


# region: recall
def recall(tp, fn):
    """Of everything that truly WAS positive, what fraction we caught.

    recall = TP / (TP + FN), a.k.a. sensitivity or the true positive rate.
    Unitless, in [0, 1]. The cost of a miss lives here — low recall means
    real cases slipped through. Undefined when there are no positives.
    """
    return tp / (tp + fn) if (tp + fn) else 0.0
# endregion


# region: specificity
def specificity(tn, fp):
    """Of everything that truly was NEGATIVE, what fraction we cleared.

    specificity = TN / (TN + FP), the true negative rate. Unitless, in
    [0, 1]. Recall's mirror image on the other class; 1 - specificity is
    the false positive rate, the x-axis of the ROC curve.
    """
    return tn / (tn + fp) if (tn + fp) else 0.0
# endregion


# region: f1
def f1(precision_val, recall_val):
    """The harmonic mean of precision and recall — one number for both.

    f1 = 2 * P * R / (P + R). Unitless, in [0, 1]. Harmonic, not
    arithmetic, on purpose: it stays near the smaller of the two, so you
    can't win F1 by acing precision while recall collapses. Zero when
    either is zero.
    """
    denom = precision_val + recall_val
    return 2 * precision_val * recall_val / denom if denom else 0.0
# endregion


# region: metrics_at
def metrics_at(y_true, y_score, threshold):
    """Every scalar metric at one decision threshold, in one call.

    Threshold the scores, count the four cells, and derive the ratios.
    Returns a dict so the animation can read a full metric snapshot per
    frame without recomputing anything.
    """
    y_pred = predict_at(y_score, threshold)
    tp, fp, fn, tn = confusion_counts(y_true, y_pred)
    p = precision(tp, fp)
    r = recall(tp, fn)
    return {
        "threshold": float(threshold),
        "tp": tp, "fp": fp, "fn": fn, "tn": tn,
        "accuracy": accuracy(tp, fp, fn, tn),
        "precision": p,
        "recall": r,
        "specificity": specificity(tn, fp),
        "f1": f1(p, r),
        "tpr": r,                       # recall == true positive rate
        "fpr": 1.0 - specificity(tn, fp),
    }
# endregion


# region: roc_curve
def roc_curve(y_true, y_score):
    """Sweep every threshold and trace (FPR, TPR) — the ROC curve.

    The thresholds that matter are the distinct score values: between two
    adjacent scores nothing changes. Walk them from high to low, so the
    curve starts at (0, 0) (threshold above every score, nothing flagged)
    and ends at (1, 1) (threshold below every score, everything flagged).
    Tied scores are grouped into a single step, which is what makes the
    trapezoidal area equal the true AUC. Returns (fpr, tpr, thresholds).
    """
    y_true = np.asarray(y_true).astype(int)
    y_score = np.asarray(y_score, dtype=float)
    P = int(np.sum(y_true == 1))
    N = int(np.sum(y_true == 0))
    thresholds = np.sort(np.unique(y_score))[::-1]   # high -> low
    fpr = [0.0]
    tpr = [0.0]
    thr = [np.inf]
    for t in thresholds:
        y_pred = (y_score >= t).astype(int)
        tp = int(np.sum((y_pred == 1) & (y_true == 1)))
        fp = int(np.sum((y_pred == 1) & (y_true == 0)))
        tpr.append(tp / P if P else 0.0)
        fpr.append(fp / N if N else 0.0)
        thr.append(float(t))
    return np.array(fpr), np.array(tpr), np.array(thr)
# endregion


# region: auc
def auc(fpr, tpr):
    """Area under the ROC curve by the trapezoidal rule.

    Integrate TPR with respect to FPR. Unitless, in [0, 1], and it has a
    clean meaning: the probability that a random positive is scored above a
    random negative. 0.5 is a coin flip, 1.0 is a perfect ranking. Because
    the curve is monotone in FPR, trapezoids give the exact area.
    """
    return float(np.trapezoid(tpr, fpr))
# endregion


# region: pr_curve
def pr_curve(y_true, y_score):
    """The precision-recall curve: precision vs recall as the threshold sweeps.

    Same threshold sweep as the ROC, read on different axes. On imbalanced
    data this curve is the honest one — it ignores the huge true-negative
    count that flatters the ROC. Returns (recall_points, precision_points).
    """
    y_true = np.asarray(y_true).astype(int)
    y_score = np.asarray(y_score, dtype=float)
    thresholds = np.sort(np.unique(y_score))[::-1]
    recalls, precisions = [], []
    for t in thresholds:
        y_pred = (y_score >= t).astype(int)
        tp, fp, fn, tn = confusion_counts(y_true, y_pred)
        recalls.append(recall(tp, fn))
        precisions.append(precision(tp, fp))
    return np.array(recalls), np.array(precisions)
# endregion

The library version

Nobody hand-rolls these in production, and you shouldn't either once you know what they mean. sklearn.metrics has every one of them, and the point of writing our own was to earn the right to trust the library's. Here it is computing the same scalars at the 0.5 threshold, plus the threshold-free AUC:

def sklearn_metrics(y_true, y_score, threshold=0.5):
    """Every scalar metric at one threshold, from sklearn.metrics.

    sklearn's confusion_matrix lays cells out as [[TN, FP], [FP-row...]] —
    rows are truth, columns are prediction, labels sorted [0, 1] — so we
    unpack it as [[TN, FP], [FN, TP]]. AUC is threshold-free, computed from
    the raw scores.
    """
    y_pred = (y_score >= threshold).astype(int)
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred, labels=[0, 1]).ravel()
    return {
        "tp": int(tp), "fp": int(fp), "fn": int(fn), "tn": int(tn),
        "accuracy": float(accuracy_score(y_true, y_pred)),
        "precision": float(precision_score(y_true, y_pred, zero_division=0)),
        "recall": float(recall_score(y_true, y_pred, zero_division=0)),
        "f1": float(f1_score(y_true, y_pred, zero_division=0)),
        "auc": float(roc_auc_score(y_true, y_score)),
    }

One layout gotcha worth flagging: sklearn's confusion_matrix puts truth on the rows and predictions on the columns with labels sorted [0, 1], so the flattened order is TN, FP, FN, TP — not the order you'd guess, and a common source of silently swapped precision and recall. The ROC comes out of roc_curve, and the area out of roc_auc_score:

def sklearn_roc(y_true, y_score):
    """sklearn's ROC curve and AUC from the raw scores."""
    fpr, tpr, thr = roc_curve(y_true, y_score)
    return fpr, tpr, thr, float(roc_auc_score(y_true, y_score))

The scores we're judging come from a plain logistic regression — the model isn't the subject here, so we keep it boring: standardize on the training stats, fit, read off predict_proba.

def train_and_score(train_df, test_df):
    """Fit logistic regression on the train split, score the test split.

    The model is not the point of this chapter — the metrics are — so we
    keep it plain: standardize the eight features on the training stats,
    fit, then read off predict_proba for the positive class on the held-out
    patients. Returns (y_true, y_score) for the test set.
    """
    Xtr = train_df[FEATURES].to_numpy(float)
    Xte = test_df[FEATURES].to_numpy(float)
    mean, std = Xtr.mean(axis=0), Xtr.std(axis=0)
    std = np.where(std == 0, 1.0, std)
    Xtr, Xte = (Xtr - mean) / std, (Xte - mean) / std

    clf = LogisticRegression(max_iter=1000, random_state=0)
    clf.fit(Xtr, train_df["Outcome"].to_numpy(int))
    y_true = test_df["Outcome"].to_numpy(int)
    y_score = clf.predict_proba(Xte)[:, 1]   # P(y = 1) column
    return y_true, y_score

Scratch versus library

If our formulas are right, every scratch number should land exactly on sklearn's. Here's the face-off across all five metrics, each one unitless on a 0–1 scale — accuracy, precision, recall, and F1 are fractions of a count at the 0.5 threshold, and AUC is the threshold-free ranking probability:

The pairs are identical to the last decimal — accuracy 0.77, precision 0.68, recall 0.61, F1 0.64, AUC 0.83 — which is the confirmation we wanted: generate_traces.py asserts the equality on every run, so if a formula ever drifts, the build fails instead of shipping a wrong number. The figures the chart draws from live in results.json, regenerated whenever the code changes.

Now read those five numbers as a group, because that's the real lesson. This model's accuracy is 0.77, which sounds respectable until you remember the majority-class baseline is 0.65 — so the model is buying you twelve points over guessing, not seventy-seven. Its recall is 0.61: at the default threshold it misses 31 of the 80 diabetics in the test set. For a screening test that number is alarming and the accuracy hid it completely. Same model, same data, and the story flips from "pretty good" to "misses two in five cases" depending only on which metric you read. That gap is the entire argument of this chapter.

Takeaways

Pick the metric that matches the cost of your errors, and pick it before you train, not after you see the numbers. If a miss is the expensive mistake — disease, fraud, safety — you optimize recall and accept the false alarms. If a false alarm is the expensive one — flagging good customers, blocking real email — you optimize precision and accept the misses. When both cost about the same and you just need one number to rank models, F1. When you haven't committed to a threshold and want to know how good the ranking is, AUC. Accuracy earns a place on that list only when the classes are balanced and the two errors cost the same, which in my experience is almost never the case anyone actually has.

The habit that saves you is to always report the confusion matrix alongside whatever headline number you quote. Every metric in this chapter is a lossy summary of those four counts, and different summaries hide different disasters — but the four counts hide nothing. A reviewer who sees TP, FP, FN, and TN can compute any metric they care about and catch the one you conveniently didn't mention. And treat any single accuracy figure on imbalanced data as a red flag until you've checked it against the majority baseline; ours looked fine at 0.77 and was quietly missing 39% of the cases it existed to find. That failure mode gets worse the rarer the positive class gets, which is where the next chapter on imbalanced classes picks the thread up — because once the positives are 1% of the data, accuracy doesn't just mislead, it lies outright.