The simplest classifier: a threshold

What this chapter covers

Before you reach for a neural network, you should know what the dumbest possible model scores on your data. That number is the bar everything else has to clear, and most of the time people never measure it. So we start the whole course here, with a classifier so simple it fits in one line: pick one number, draw a line, call everyone above the line sick and everyone below it healthy.

We build it by hand in NumPy, one function at a time, then let scikit-learn build the exact same thing and check that the numbers agree. Along the way you get to watch the "training" happen — the cutoff slides across real patient data and you see the accuracy rise and fall with it. No magic, no gradients yet, just a search over one number. Get this and the rest of the course is variations on a theme.

The data is the Pima Indians Diabetes set: 768 patients, eight measurements each, and a label that says whether they were diagnosed with diabetes. We'll only use one of the eight columns for now — blood glucose.

A bit of history

Thresholding a single measurement is older than machine learning and older than computers. Medicine has run on it for a century: a fasting glucose over some cutoff means diabetes, a blood pressure over another means hypertension. The World Health Organization still defines diabetes with a fixed glucose line. Somebody picked that number by looking at outcomes, which is exactly what we're about to do, just with a computer doing the looking.

On the statistics side, Ronald Fisher formalized the idea in 1936 with linear discriminant analysis: project the data onto one axis, then split it with a threshold. And in machine learning the one-feature threshold has a name — a decision stump, a decision tree pruned down to a single split. It looks like a toy, but it turned out to matter. Freund and Schapire's AdaBoost (1997) stacks hundreds of these stumps into one of the strongest classifiers of its era. Every decision tree you'll ever train also starts its life as a stump: the root split is exactly this. So the thing we build today is both the weakest useful model and the atom that bigger models are made of.

The intuition

Look at a chart of glucose readings, colored by who actually had diabetes. The two groups overlap, but they don't sit on top of each other — the diabetic readings lean high. If they leaned high perfectly, with no overlap, one vertical line would separate them and we'd be done. They don't, so no single line is perfect. The job is to find the line that's wrong the least often.

That's the entire model. One feature on the x-axis, one vertical line, a guess of "diabetic" for everything to the right of it. "Training" means trying lines until you find the best one. There's nothing else to it, and that's the point — everything harder in this course is a smarter way to draw a boundary when one straight line in one dimension isn't enough.

Here's the real data. Glucose readings for both groups, drawn as smoothed distributions so you can see the overlap and the lean:

The blue hump (no diabetes) peaks around 100, the purple one (diabetes) sits well to the right. Where they overlap is where a threshold has to make hard calls, and where our accuracy leaks.

The math

Let $g_i$ be patient $i$'s glucose reading and $\theta$ the cutoff. The prediction is a single indicator:

One if glucose is at or above the cutoff, zero otherwise. Accuracy is just how often that matches the true label $y_i$ across all $N$ patients:

Training is picking the cutoff that maximizes it. We only search the range where glucose actually varies, 75 to 200:

That's the whole learning algorithm — an argmax over one integer. To see where the accuracy comes from and where it goes, we count outcomes in a 2×2 confusion matrix:

Row $j$ is what we predicted, column $k$ is the truth. The diagonal is the patients we got right. Off the diagonal are the two ways to be wrong, and they are not the same kind of wrong — one is a missed diabetic, the other a false alarm. Hold that thought; it comes back when we watch the matrix move.

What it's good at, what it isn't

The good side is real. It's interpretable to the point of being obvious — the model is one number a doctor can read off and argue with. It trains in milliseconds, needs almost no data to estimate one cutoff, and it's impossible to overfit a single threshold. As a baseline it's honest: if your fancy model can't beat one line on one feature, the fancy model is not learning anything.

The bad side is just as real. It sees one feature and ignores the other seven, so any signal that lives in a combination of measurements is invisible to it. The boundary is axis-aligned and straight — one cut, perpendicular to one axis, no curves, no interactions. On data where the classes spiral around each other or depend on two features together, a stump is helpless. It's a floor, not a ceiling. The rest of the course is about lifting off that floor.

Build it, one function at a time

The whole thing is six short functions. Each one does exactly one job, and each builds on the last. This is the same order I'd write it at a terminal: smallest testable piece first, then wrap it.

The decision rule itself. Everything else is bookkeeping around this line:

def simple_pred_vec(g, theta):
    """The entire decision rule: diabetic when glucose >= theta.

    Both arguments broadcast, so g can be one reading or a whole column,
    and theta one cutoff or a column of candidate cutoffs.
    """
    return g >= theta

Because that comparison broadcasts, the same function handles one reading against one cutoff, or a whole column of readings against a whole column of candidate cutoffs. We lean on that hard in a minute. First, apply it across a dataframe of patients:

def simple_pred_vec(g, theta):
    """The entire decision rule: diabetic when glucose >= theta.

    Both arguments broadcast, so g can be one reading or a whole column,
    and theta one cutoff or a column of candidate cutoffs.
    """
    return g >= theta

Reshaping theta to a column and glucose to a row turns one comparison into a full (T, N) grid — every candidate cutoff scored against every patient at once. Now accuracy is one line: compare that grid to the truth and average across patients.

def simple_acc(df, theta):
    """Fraction of patients the rule gets right, for each candidate theta."""
    y = df["Outcome"].to_numpy()          # true labels, shape (N,)
    pred = simple_pred(df, theta)         # shape (T, N)
    return (pred == y).mean(axis=1)       # shape (T,)

With accuracy in hand, "training" is a search. Here it is the obvious way, with a loop over every integer cutoff:

def best_theta_loopy(df):
    """Brute force: try every integer cutoff from 75 to 200, keep the best.

    Ties go to the first (lowest) cutoff that reaches the best accuracy.
    """
    best_t, best_a = None, -1.0
    for t in range(75, 201):
        a = float(simple_acc(df, t)[0])
        if a > best_a:
            best_t, best_a = t, a
    return best_t, best_a

That works and it's readable, and for 126 cutoffs it's plenty fast. But we already built simple_acc to score every cutoff in one shot, so the loop is doing work NumPy would rather do itself. Same answer, no Python loop:

def best_theta_loopy(df):
    """Brute force: try every integer cutoff from 75 to 200, keep the best.

    Ties go to the first (lowest) cutoff that reaches the best accuracy.
    """
    best_t, best_a = None, -1.0
    for t in range(75, 201):
        a = float(simple_acc(df, t)[0])
        if a > best_a:
            best_t, best_a = t, a
    return best_t, best_a

Both versions return the same cutoff and the same accuracy — argmax breaks ties on the first maximum, exactly like the loop keeping the first best it sees. The vectorized one is the version you keep. Getting comfortable turning a loop into a broadcast is half of what makes NumPy worth using, and it's a habit that pays off in every later chapter.

Last piece: accuracy is one number, and one number hides how you're wrong. The confusion matrix splits it open.

def simple_confusion(df, theta):
    """The 2x2 confusion matrix for the rule at a given cutoff.

    M[i, j] counts patients *predicted* i whose *true* label is j
    (0 = no diabetes, 1 = diabetes). So M[0, 1] is the missed diabetics
    (false negatives) and M[1, 0] the false alarms (false positives).
    """
    y = df["Outcome"].to_numpy()
    pred = simple_pred(df, theta)[0].astype(int)
    M = np.zeros((2, 2), dtype=int)
    for i in range(2):
        for j in range(2):
            M[i, j] = int(((pred == i) & (y == j)).sum())
    return M

Watch it work

This is the search itself, running on a small, clean toy set so you can actually see it — about 60 patients, two groups, one dimension. The orange line is the current cutoff. Points get a red ring the moment the cutoff misclassifies them. The lower panel traces accuracy as the cutoff moves, so you watch the peak arrive.

Press play. This is best_theta_loopy with the loop slowed down to human speed — every frame is one real iteration, scoring one real cutoff. Reset and run it again as many times as you like.

Watch what happens at the extremes. With the cutoff far left, everyone is predicted diabetic, so every healthy point wears a red ring. Far right, nobody is, so the sick points light up instead. The accuracy curve is low at both ends and humps in the middle, and the top of that hump is the cutoff we keep.

Now the same search on the real 768-patient data, seen through the confusion matrix. Same sweep, but instead of dots this shows the four counts — correct healthy, correct diabetic, and the two kinds of mistake — updating live as the cutoff moves.

This is the trade-off that thresholding can't escape, and it's worth sitting with. Slide the cutoff low and you catch almost every diabetic — the missed cases (top-right cell) drop toward zero — but you also flag a pile of healthy people (bottom-left cell balloons). Slide it high and the false alarms vanish while missed diabetics pile up. There is no setting that zeroes both. Where you put the line is a decision about which mistake you'd rather make, and in medicine that's not a math question. Accuracy alone won't tell you; the matrix will.

Here's the accuracy curve over the full dataset, with the winning cutoff marked. The peak is broad and a little flat on top, which is the visual way of saying a stump is a blunt instrument — a whole band of cutoffs scores about the same.

The full implementation

Six functions, no library, top to bottom. This is the file the animations above actually ran:

"""The simplest classifier: a glucose threshold, built from scratch.

Predict diabetic when a patient's blood glucose is at or above a cutoff
theta; the whole "training" is a brute-force search for the cutoff with the
highest accuracy. Pure NumPy — no ML library anywhere in this file.

Every function below appears in the chapter one step at a time (the
`# region:` markers are what the book's include directives pull in).
"""

import numpy as np
import pandas as pd


# region: simple_pred_vec
def simple_pred_vec(g, theta):
    """The entire decision rule: diabetic when glucose >= theta.

    Both arguments broadcast, so g can be one reading or a whole column,
    and theta one cutoff or a column of candidate cutoffs.
    """
    return g >= theta
# endregion


# region: simple_pred
def simple_pred(df, theta):
    """Apply the rule to every patient in the dataframe.

    theta may be a scalar or a vector of T candidate cutoffs. The result is
    a (T, N) boolean array: one row of predictions per candidate, produced
    by a single broadcast comparison — no loops.
    """
    g = df["Glucose"].to_numpy()                              # shape (N,)
    thetas = np.atleast_1d(np.asarray(theta)).reshape(-1, 1)  # shape (T, 1)
    return simple_pred_vec(g, thetas)                         # shape (T, N)
# endregion


# region: simple_acc
def simple_acc(df, theta):
    """Fraction of patients the rule gets right, for each candidate theta."""
    y = df["Outcome"].to_numpy()          # true labels, shape (N,)
    pred = simple_pred(df, theta)         # shape (T, N)
    return (pred == y).mean(axis=1)       # shape (T,)
# endregion


# region: best_theta_loopy
def best_theta_loopy(df):
    """Brute force: try every integer cutoff from 75 to 200, keep the best.

    Ties go to the first (lowest) cutoff that reaches the best accuracy.
    """
    best_t, best_a = None, -1.0
    for t in range(75, 201):
        a = float(simple_acc(df, t)[0])
        if a > best_a:
            best_t, best_a = t, a
    return best_t, best_a
# endregion


# region: best_theta
def best_theta(df):
    """The same search with the loop pushed into NumPy.

    One (126, N) comparison scores every cutoff at once; argmax picks the
    winner. np.argmax returns the FIRST maximum, so ties break exactly like
    the loop above.
    """
    thetas = np.arange(75, 201)
    acc = simple_acc(df, thetas)
    i = int(np.argmax(acc))
    return int(thetas[i]), float(acc[i])
# endregion


# region: simple_confusion
def simple_confusion(df, theta):
    """The 2x2 confusion matrix for the rule at a given cutoff.

    M[i, j] counts patients *predicted* i whose *true* label is j
    (0 = no diabetes, 1 = diabetes). So M[0, 1] is the missed diabetics
    (false negatives) and M[1, 0] the false alarms (false positives).
    """
    y = df["Outcome"].to_numpy()
    pred = simple_pred(df, theta)[0].astype(int)
    M = np.zeros((2, 2), dtype=int)
    for i in range(2):
        for j in range(2):
            M[i, j] = int(((pred == i) & (y == j)).sum())
    return M
# endregion


def load_data(path="../data/diabetes.csv"):
    """Pima Indians Diabetes dataset: 768 patients, 8 features, Outcome."""
    return pd.read_csv(path)

The library version

Nobody hand-rolls a threshold search in practice, and you shouldn't either once you understand it. A decision stump — a tree with max_depth=1 — is the same model, and scikit-learn will fit one for you:

def stump_glucose(train_df, test_df):
    """A depth-1 tree on the single Glucose column — the library twin of
    our hand-built rule. Returns the learned cutoff and test accuracy."""
    stump = DecisionTreeClassifier(max_depth=1, random_state=0)
    stump.fit(train_df[["Glucose"]], train_df["Outcome"])
    cutoff = float(stump.tree_.threshold[0])
    acc = float(stump.score(test_df[["Glucose"]], test_df["Outcome"]))
    return cutoff, acc

One difference under the hood: we searched for the cutoff that maximizes accuracy, sklearn searches for the one that minimizes Gini impurity. Different objective, but on this data they land essentially on the same line. The other difference is more interesting. Give the stump all eight features instead of just glucose and it will pick which feature to split on by itself:

def stump_all_features(train_df, test_df):
    """Give the stump all 8 features and let it choose the split itself.
    Returns the feature it picked, the cutoff, and test accuracy."""
    stump = DecisionTreeClassifier(max_depth=1, random_state=0)
    stump.fit(train_df[FEATURES], train_df["Outcome"])
    feature = FEATURES[int(stump.tree_.feature[0])]
    cutoff = float(stump.tree_.threshold[0])
    acc = float(stump.score(test_df[FEATURES], test_df["Outcome"]))
    return feature, cutoff, acc

That's the stump earning its keep as a diagnostic. The feature it chooses is the single most predictive measurement in the whole set, ranked for free. On this data it picks glucose — which is the quiet confirmation that our one-feature hunch was the right feature all along.

Scratch versus library

Split the data 70/30, fit on the training half, score on the held-out half — both our search and sklearn's stump. Here's the face-off:

The bars sit on top of each other, which is the result you want. Our hand-written argmax and sklearn's Gini stump reach the same test accuracy with cutoffs a hair apart, and handing the stump all eight features doesn't move the number — it just reconfirms glucose was the one that mattered. The exact figures the chart is drawn from live in results.json, regenerated whenever the code changes, so the prose and the picture can't drift from what the code actually does.

Two honest caveats about that accuracy. On the full dataset the best cutoff scores about 75%, which sounds decent until you notice that guessing "no diabetes" for everyone scores about 65% — the classes are lopsided, so the floor is higher than 50%. And the accuracy on data you trained on is optimistic; that's why the face-off above uses a held-out split. Both of these are lessons the stump teaches cheaply that you'll pay for expensively if you learn them later on a bigger model.

Takeaways

Fit the stump first. Every time. Before the gradient boosting, before the network, before any of it — spend the two seconds to learn what one line on one feature scores, because that's the number your real model has to beat to justify its existence. I've watched too many pipelines ship a complicated model that a single threshold would have matched, and nobody checked because nobody measured the floor.

The stump pays you three ways. It's your honest baseline. It's a free feature ranker — hand it every column and the split it picks is your most predictive signal, no extra code. And it's the atom the rest of the course assembles: boost a pile of stumps and you get AdaBoost, stack splits into a tree and you get a forest. Reach for something heavier when one straight cut in one dimension leaves accuracy on the table — when the signal lives in a combination of features, or the boundary needs to curve. That's most real problems, which is why the course keeps going. But it starts here, because the floor is where you find out whether you're actually climbing.