When accuracy lies: imbalanced classes

What this chapter covers

The previous chapter showed accuracy hiding a mediocre model at 35% prevalence. This one shows it lying outright. Our test set is 4.5% positive, and a model that predicts the majority class for all 600 rows scores 0.955 accuracy while finding exactly none of the cases it exists to find. That is not a broken model. That is the arithmetic of a rare class, and no amount of training fixes it if you keep reading accuracy.

So this chapter is about what to do instead. We build the three classic rebalancers by hand in NumPy — random oversampling of the minority, random undersampling of the majority, and the balanced class weights that reweight the loss without touching the data — then hand each one to the same logistic regression and watch what changes. What changes is the decision boundary, and the story it tells is the whole point: as we apply each fix the boundary swings off the majority and wraps the tiny minority cluster, recall climbs from catching one case in five to catching nearly all of them, and accuracy barely flinches because there was never much accuracy in it to begin with. Every number on this page is judged with the precision, recall, F1, and AUC from the metrics chapter, because on this data those are the only numbers that mean anything.

A bit of history

The problem is old and the name for it is blunt. By the late 1990s enough people had been burned by it that the machine-learning community started holding workshops specifically about learning from imbalanced data — the AAAI workshop in 2000, an ICML one in 2003 — because the standard algorithms, all of them tuned to minimize error rate, were quietly optimizing for the majority and declaring victory. Foster Provost's short 2000 note, "Machine Learning from Imbalanced Data Sets," put the complaint plainly: accuracy assumes equal error costs and balanced classes, and when neither holds it is worse than useless because it actively rewards the wrong model. Somewhere in that period the phrase "accuracy paradox" stuck — the paradox being that throwing away the classifier and always guessing the majority class can raise your accuracy.

The fixes have their own lineage. Reweighting the classes is as old as cost-sensitive learning. Undersampling and oversampling were the obvious first moves, and by 1997 Miroslav Kubat and Stan Matwin were already pointing out that naively dropping majority examples throws away information and naively copying minority ones invites overfitting. That tension is exactly what Nitesh Chawla's 2002 SMOTE paper tried to escape by synthesizing new minority points between existing ones instead of duplicating them, and SMOTE went on to become one of the most cited papers in the field. We stop short of SMOTE here — it needs its own chapter — but random oversampling is its honest, dumb ancestor, and you should meet the ancestor first.

The intuition

Here is the data, and the whole difficulty is visible in one picture: a dense cyan cloud of the majority class and, off to one side, a small purple smudge of the minority.

Count the purple. There are 107 minority points in 2000, about 5%, and they sit half-buried in the edge of the majority cloud rather than cleanly apart. Now imagine a classifier that only cares about being right as often as possible. The cheapest way to be right on this data is to color the entire plane cyan. It costs you every purple point, but purple is 5% of the plane, so you are still right 95% of the time — and a model trained to minimize error will drift toward exactly that, because the 100 minority mistakes barely register against the 1900 majority points it gets for free. The boundary that minimizes total error and the boundary that actually finds the minority are two different lines, and the entire chapter is about dragging the model from the first to the second.

The math

Start with the baseline you have to beat. Split the labels into classes, count each, and always predict the biggest. Its accuracy is the majority fraction:

where $n_c$ is the number of rows in class $c$ and $n$ the total. That is the whole trap in one line. The number is large exactly when the data is imbalanced, so on a 95/5 split the do-nothing model starts at 0.95, and any real model whose accuracy you quote has to be read against that floor, not against zero. Report 0.96 without mentioning that a brick scores 0.955 and you have said nothing.

Two of the fixes change the data; the third changes the loss. Ordinary logistic regression minimizes the mean cross-entropy, which treats every row as equally important:

Class weighting slips a per-class multiplier in front of each term, so a mistake on a rare-class row costs more than a mistake on a common one:

The balanced choice of weights sets each class's weight inversely to its size:

with $k$ the number of classes. On our training split that makes the minority weight 8.75 and the majority weight 0.53 — a minority error hurts the loss about sixteen times as much as a majority error, which is roughly the imbalance ratio, and that is the point. Oversampling reaches the same place by copying minority rows until they are as numerous as the majority; a row that appears sixteen times contributes sixteen identical terms to the sum, which is arithmetically almost the same as multiplying its one term by sixteen. The three fixes are three routes to the same idea: make the minority matter.

What each fix is good at, and what it isn't

Undersampling is the cheapest thing you can do and often the first thing I try, because a balanced training set that is a tenth the size trains in a tenth the time and you find out fast whether the signal is even there. Its flaw is that it throws away real majority data — on our data it discards over a thousand rows to match eighty — and everything you discard is information the model no longer sees. Oversampling keeps all the data but pays in the other direction: the duplicated minority rows are literally the same points, so a flexible model can memorize them, and you have inflated your confidence in a handful of examples without adding any actual variety. Neither invents information that was not in the eighty minority rows to begin with.

Class weighting is the one I reach for by default, because it is a single argument, it touches no data, it keeps every row, and for a linear model it is mathematically about the same as balanced oversampling without the memorization risk or the bookkeeping. What none of the three do is make the model better at telling the classes apart. They move the operating point — where the boundary sits — not the model's underlying ability to rank a minority point above a majority one. You will see that in a moment when the AUC refuses to move while everything else does, and it is the most important thing on the page.

The data

The dataset is synthetic on purpose, generated once with scikit-learn's make_classification at weights=[0.95, 0.05] and committed as a snapshot so every number here is reproducible. Two features, so we can watch the boundary in the plane; 2000 rows, of which 107 are the minority; a class_sep low enough that the minority sits partly inside the majority rather than cleanly apart, because a cleanly separated minority would be caught by the naive model and there would be nothing to teach. We split it 70/30 into 1400 training and 600 test rows. The training split carries 80 minority rows, the test split 27.

Everything downstream obeys one rule that this chapter will repeat until it is boring: we resample and reweight the training set only. The test set keeps its real 4.5% prevalence from start to finish, because the test set exists to estimate performance in the world, and the world is imbalanced. Rebalance your test set and you are grading the model on a distribution it will never see.

Build it, one function at a time

Start with the baseline, because it is the number every fix is measured against. The majority-class accuracy is just the biggest class count over the total — the accuracy of a model that has learned nothing:

def majority_class(y):
    """The label that appears most often — what a lazy model would always guess."""
    y = np.asarray(y).astype(int)
    return int(np.argmax(np.bincount(y)))


def majority_baseline_accuracy(y):
    """Accuracy of the model that predicts the majority class for everyone.

    It equals the majority-class fraction: if 95% of rows are class 0, always
    answering 0 is right 95% of the time while learning nothing. Unitless, in
    [0, 1]. This is the bar any classifier has to clear before its accuracy
    means a single thing, and on imbalanced data it sits embarrassingly high.
    """
    y = np.asarray(y).astype(int)
    counts = np.bincount(y)
    return float(counts.max() / counts.sum())

Now the first real fix. Random oversampling keeps every majority row and draws minority rows with replacement until the two classes are the same size. The seeded generator is not decoration — resampling is randomness inside your training pipeline, and an unseeded pipeline gives you a different model every run and no way to reproduce a result:

def random_oversample(X, y, seed=0):
    """Random oversampling: duplicate minority rows until the classes match.

    Keep every majority row; draw minority rows WITH replacement until there
    are as many of them as the majority class. Nothing new is invented — the
    same minority points just show up several times, so they weigh more in the
    fit. Seeded, so the draw is identical every run. Returns the balanced,
    shuffled (X, y). Resample the TRAINING set only — never the test set.
    """
    X = np.asarray(X, dtype=float)
    y = np.asarray(y).astype(int)
    rng = np.random.default_rng(seed)
    counts = np.bincount(y)
    n_target = counts.max()
    parts_X, parts_y = [], []
    for c in range(len(counts)):
        idx = np.flatnonzero(y == c)
        if len(idx) == 0:
            continue
        take = idx if len(idx) >= n_target else rng.choice(idx, size=n_target, replace=True)
        parts_X.append(X[take])
        parts_y.append(y[take])
    Xb = np.vstack(parts_X)
    yb = np.concatenate(parts_y)
    perm = rng.permutation(len(yb))
    return Xb[perm], yb[perm]

Undersampling is the mirror image: keep every minority row, and draw majority rows without replacement down to the minority count. Same shape of code, opposite sacrifice — this one balances by deletion instead of duplication:

def random_undersample(X, y, seed=0):
    """Random undersampling: drop majority rows until the classes match.

    Keep every minority row; draw majority rows WITHOUT replacement down to the
    minority count. Fast and it fixes the balance, but it throws real data on
    the floor — the discarded majority rows are gone from the fit. Seeded and
    repeatable. Returns the balanced, shuffled (X, y). Training set only.
    """
    X = np.asarray(X, dtype=float)
    y = np.asarray(y).astype(int)
    rng = np.random.default_rng(seed)
    counts = np.bincount(y)
    n_target = counts[counts > 0].min()
    parts_X, parts_y = [], []
    for c in range(len(counts)):
        idx = np.flatnonzero(y == c)
        if len(idx) == 0:
            continue
        take = idx if len(idx) <= n_target else rng.choice(idx, size=n_target, replace=False)
        parts_X.append(X[take])
        parts_y.append(y[take])
    Xb = np.vstack(parts_X)
    yb = np.concatenate(parts_y)
    perm = rng.permutation(len(yb))
    return Xb[perm], yb[perm]

The third fix never touches the data. It computes a weight per class, big for the rare one and small for the common one, using the balanced rule from the math section. This returns the same numbers scikit-learn computes for class_weight='balanced', which the trace script asserts on every run:

def balanced_class_weights(y):
    """Per-class weights that make the loss treat the classes as equal.

        w_c = n / (k * n_c)

    total rows n, over the number of classes k times the count of class c. The
    rare class gets a large weight, the common class a small one, so a mistake
    on a minority row costs proportionally more in the loss than a mistake on a
    majority row. No data is copied or dropped — only the penalty changes. This
    is exactly scikit-learn's class_weight='balanced'. Returns {class: weight}.
    """
    y = np.asarray(y).astype(int)
    counts = np.bincount(y)
    k = int(np.sum(counts > 0))
    n = int(counts.sum())
    return {c: float(n / (k * counts[c])) for c in range(len(counts)) if counts[c] > 0}

And because a classifier wants a weight per row rather than per class, one small helper spreads the class weights across the rows — the bridge from the formula to the argument fit actually takes:

def sample_weights(y, class_weight):
    """Spread a per-class weight dict into a per-row weight vector.

    Feed this to a classifier's sample_weight to reproduce class weighting by
    hand: every row inherits its class's weight. It's the bridge that turns the
    balanced_class_weights dict into the argument sklearn's fit() wants.
    """
    y = np.asarray(y).astype(int)
    return np.array([class_weight[int(c)] for c in y], dtype=float)

That is the entire toolkit: a baseline to beat and three ways to make the minority count. None of it trains a model. The model is a stock logistic regression; these functions just decide what it sees and how much each row weighs.

Watch it work

This is the chapter in one animation. Each frame applies one fix and refits, then draws the resulting decision boundary over the same 2000 points — the orange line, with everything below it predicted minority — and reports the four metrics on the held-out test set underneath. The data never changes; only the boundary and the bars do.

Frame one is the naive fit, and it is the accuracy paradox drawn to scale. The boundary stands almost vertical at the far edge of the plane, slicing off a thin sliver and calling everything else majority. Accuracy reads 0.96, which looks like a working model until you find the recall: 0.19. Out of 27 real minority cases in the test set it catches five. The 0.96 accuracy is buying you a model that misses more than four out of five of the cases it was built to find, and if you only ever printed accuracy you would ship it.

Step forward and watch the fixes work. Oversampling rotates the boundary flat and shoves it down over the minority cluster; undersampling and class weighting land on almost the same line. Recall jumps to 0.96, then 1.0 — the model now catches every minority case in the test set. And accuracy? It settles around 0.82. It moved, but look at the scale of the two moves: recall multiplied by five while accuracy slipped four points, and those four points were never worth anything because they were four points above a 0.955 floor. That is the trade these fixes make, stated honestly. There is a cost, and it is precision, which collapses from 0.71 to 0.20 as the flattened boundary sweeps in a crowd of majority points along with the minority. F1, which has to answer to both, edges up from 0.29 to 0.33 rather than soaring, because you bought recall by spending precision.

Now watch the one number that stays still. The AUC sits at 0.93 in the naive frame and 0.94 in every fixed frame — it does not move, because none of these fixes made the model any better at ranking a minority point above a majority one. They moved the threshold, effectively, not the model. That is the deepest lesson in the animation: the naive model was never bad at separating the classes, it was bad at where it drew the line, and resampling and reweighting are elaborate ways of redrawing the line. The last frame proves it — same naive model, no retraining, just the decision threshold moved from 0.50 down to 0.25, and the boundary slides over in parallel to a much better operating point at F1 0.39, the best on the page, for free.

The full implementation

The whole toolkit, pure NumPy, no resampling library anywhere. This is the file the animation ran:

"""Fixing imbalanced classes, from scratch.

When one class is rare, accuracy stops meaning anything and the usual fixes
are all about putting weight back on the minority. This file is the three
classic rebalancers, built in pure NumPy so nothing is hidden:

  - the majority-class baseline (the number every real model must beat),
  - random OVERSAMPLING of the minority,
  - random UNDERSAMPLING of the majority,
  - the balanced CLASS WEIGHTS that reweight the loss instead of the data.

No imbalanced-learn, no resampling library — just index juggling with a
seeded NumPy generator so every draw is repeatable. The classifier itself is
scikit-learn's LogisticRegression (see library.py); these functions produce
the training sets and weights we hand it. Convention: label 1 is the minority,
the positive class we are trying to catch; label 0 is the majority.
"""

import numpy as np


# region: majority_baseline
def majority_class(y):
    """The label that appears most often — what a lazy model would always guess."""
    y = np.asarray(y).astype(int)
    return int(np.argmax(np.bincount(y)))


def majority_baseline_accuracy(y):
    """Accuracy of the model that predicts the majority class for everyone.

    It equals the majority-class fraction: if 95% of rows are class 0, always
    answering 0 is right 95% of the time while learning nothing. Unitless, in
    [0, 1]. This is the bar any classifier has to clear before its accuracy
    means a single thing, and on imbalanced data it sits embarrassingly high.
    """
    y = np.asarray(y).astype(int)
    counts = np.bincount(y)
    return float(counts.max() / counts.sum())
# endregion


# region: oversample
def random_oversample(X, y, seed=0):
    """Random oversampling: duplicate minority rows until the classes match.

    Keep every majority row; draw minority rows WITH replacement until there
    are as many of them as the majority class. Nothing new is invented — the
    same minority points just show up several times, so they weigh more in the
    fit. Seeded, so the draw is identical every run. Returns the balanced,
    shuffled (X, y). Resample the TRAINING set only — never the test set.
    """
    X = np.asarray(X, dtype=float)
    y = np.asarray(y).astype(int)
    rng = np.random.default_rng(seed)
    counts = np.bincount(y)
    n_target = counts.max()
    parts_X, parts_y = [], []
    for c in range(len(counts)):
        idx = np.flatnonzero(y == c)
        if len(idx) == 0:
            continue
        take = idx if len(idx) >= n_target else rng.choice(idx, size=n_target, replace=True)
        parts_X.append(X[take])
        parts_y.append(y[take])
    Xb = np.vstack(parts_X)
    yb = np.concatenate(parts_y)
    perm = rng.permutation(len(yb))
    return Xb[perm], yb[perm]
# endregion


# region: undersample
def random_undersample(X, y, seed=0):
    """Random undersampling: drop majority rows until the classes match.

    Keep every minority row; draw majority rows WITHOUT replacement down to the
    minority count. Fast and it fixes the balance, but it throws real data on
    the floor — the discarded majority rows are gone from the fit. Seeded and
    repeatable. Returns the balanced, shuffled (X, y). Training set only.
    """
    X = np.asarray(X, dtype=float)
    y = np.asarray(y).astype(int)
    rng = np.random.default_rng(seed)
    counts = np.bincount(y)
    n_target = counts[counts > 0].min()
    parts_X, parts_y = [], []
    for c in range(len(counts)):
        idx = np.flatnonzero(y == c)
        if len(idx) == 0:
            continue
        take = idx if len(idx) <= n_target else rng.choice(idx, size=n_target, replace=False)
        parts_X.append(X[take])
        parts_y.append(y[take])
    Xb = np.vstack(parts_X)
    yb = np.concatenate(parts_y)
    perm = rng.permutation(len(yb))
    return Xb[perm], yb[perm]
# endregion


# region: class_weights
def balanced_class_weights(y):
    """Per-class weights that make the loss treat the classes as equal.

        w_c = n / (k * n_c)

    total rows n, over the number of classes k times the count of class c. The
    rare class gets a large weight, the common class a small one, so a mistake
    on a minority row costs proportionally more in the loss than a mistake on a
    majority row. No data is copied or dropped — only the penalty changes. This
    is exactly scikit-learn's class_weight='balanced'. Returns {class: weight}.
    """
    y = np.asarray(y).astype(int)
    counts = np.bincount(y)
    k = int(np.sum(counts > 0))
    n = int(counts.sum())
    return {c: float(n / (k * counts[c])) for c in range(len(counts)) if counts[c] > 0}
# endregion


# region: sample_weights
def sample_weights(y, class_weight):
    """Spread a per-class weight dict into a per-row weight vector.

    Feed this to a classifier's sample_weight to reproduce class weighting by
    hand: every row inherits its class's weight. It's the bridge that turns the
    balanced_class_weights dict into the argument sklearn's fit() wants.
    """
    y = np.asarray(y).astype(int)
    return np.array([class_weight[int(c)] for c in y], dtype=float)
# endregion

The library version

The resamplers and the weight formula are ours; the classifier and the scoring are scikit-learn's. The model is fit the same way every time — the only thing that changes is what data or what weights it gets — so the four methods are a fair comparison rather than four different models:

def fit_logreg(X, y, class_weight=None, sample_weight=None):
    """Plain logistic regression, optionally reweighted.

    class_weight passes straight through to sklearn (e.g. 'balanced');
    sample_weight lets us feed the per-row weights from impl.sample_weights.
    Leave both None for the naive fit on whatever (X, y) you pass — that same
    call trains the oversampled and undersampled models, just on rebalanced
    data. random_state pins the solver so the boundary is reproducible.
    """
    clf = LogisticRegression(max_iter=2000, class_weight=class_weight, random_state=0)
    clf.fit(X, y, sample_weight=sample_weight)
    return clf

Scoring is straight out of sklearn.metrics: accuracy over all rows, precision and recall and F1 on the minority, and the threshold-free AUC from the raw scores. These are the library's numbers, which is the point — we implemented the resampling, not the metrics, so we let the library judge:

def predict_scores(clf, X):
    """The model's P(y = 1) — its minority score — for each row."""
    return clf.predict_proba(X)[:, 1]


def score_all(y_true, y_score, threshold=0.5):
    """Every headline metric at one threshold, from sklearn.metrics.

    accuracy is over all rows; precision, recall and F1 are on the minority
    (positive) class; AUC is threshold-free, read from the raw scores. All five
    are unitless in [0, 1]. On imbalanced data accuracy and AUC lean on the
    easy majority, while precision/recall/F1 report on the class you care about.
    """
    y_pred = (np.asarray(y_score) >= threshold).astype(int)
    return {
        "accuracy": float(accuracy_score(y_true, y_pred)),
        "precision": float(precision_score(y_true, y_pred, zero_division=0)),
        "recall": float(recall_score(y_true, y_pred, zero_division=0)),
        "f1": float(f1_score(y_true, y_pred, zero_division=0)),
        "auc": float(roc_auc_score(y_true, y_score)),
    }

The last frame's free lunch comes from one more helper: sweep the thresholds on the training scores and keep the one with the best F1. Tuned on train, reported on test — the same discipline as resampling, for the same reason:

def best_f1_threshold(y_true, y_score):
    """The cutoff on the minority score that maximizes F1.

    Choose it on the data you pass and always pass TRAIN scores — picking a
    threshold on the test set is the same leakage sin as resampling it. Sweeps
    the distinct score values and keeps the F1 winner. Returns the threshold.
    """
    y_true = np.asarray(y_true).astype(int)
    y_score = np.asarray(y_score, dtype=float)
    best_t, best_f1 = 0.5, -1.0
    for t in np.unique(y_score):
        f = f1_score(y_true, (y_score >= t).astype(int), zero_division=0)
        if f > best_f1:
            best_f1, best_t = f, float(t)
    return best_t

The four fixes, side by side

Here are all four methods across all five metrics, every value unitless on a 0–1 scale, straight from results.json. Read it column by column and the shape of the whole chapter is in one chart:

The accuracy cluster barely moves — 0.96 for the naive model, around 0.82 for the three fixes — and if accuracy were your yardstick you would conclude the fixes made the model slightly worse and walk away. The recall cluster tells the opposite story: 0.19 naive, then 0.96, 1.0, 1.0. Same four models, and the two metrics disagree about which is best because they are measuring different things — accuracy is dominated by the 95% majority the model was already acing, recall reports only on the 5% you actually care about. Precision moves the other way, from 0.71 down to about 0.20, the bill for that recall. And AUC sits flat near 0.94 across all four, confirming what the animation showed: these are the same ranker at different operating points, not four different qualities of model.

One honest caveat this chart makes concrete. F1 hardly improves, and on this data the blunt "balance it and predict at 0.50" recipe overshoots — it maximizes recall at a precision so low that F1 stalls. The threshold-tuned operating point from the last animation frame, which results.json records at accuracy 0.94, recall 0.41, precision 0.38 and F1 0.39, beats every bar here on F1 without retraining anything. Resampling is not the only lever, and it is rarely the sharpest one.

Takeaways

Never report a bare accuracy on imbalanced data. The first thing to compute on any classification problem is the class balance, and the second is the majority baseline, because until you know that a brick scores 0.955 you cannot read a 0.96 as anything. If your accuracy is close to the majority fraction, your model may have learned nothing, and accuracy is the one number that will never tell you.

Pick the metric that matches the cost of the rare class before you touch a fix. If a miss is the expensive mistake — fraud, disease, a failing part — you are optimizing recall and you will accept the precision hit these fixes cost. If a false alarm is expensive, you cannot just balance-and-forget, because that is exactly what tanks precision here. F1 when you genuinely need both, and always alongside the confusion matrix, never instead of it.

Resample the training set only, never the test set, and seed the resampling. The test set is your estimate of the real, imbalanced world; rebalancing it grades the model on a fantasy. And resampling is randomness inside your pipeline, so an unseeded run is not reproducible — a wrong result you cannot recreate is worse than no result.

Reach for class weights first. It is one argument, it keeps all your data, it avoids the memorization risk of oversampling and the information loss of undersampling, and for a linear model it lands in almost the same place as balanced oversampling. Then remember the cheapest lever of all: the threshold. The AUC that would not move through all three resampling methods is telling you the model already ranks the classes fine, and the honest fix is often not to rebuild the training set but to move the decision threshold to the point that matches what a miss and a false alarm actually cost you. Calibrate the threshold to the business, not to 0.50, and let the metric that matches your rare-class cost decide where it lands.