What machine learning is

What this chapter covers

Machine learning is one idea worn many ways: instead of writing the rules by hand, you write down what a good rule would look like and let a computer search for the one that fits your data best. That's it. Every chapter after this is a different answer to the same three questions — what family of rules am I searching, how do I score a rule against the data, and how do I find the best one — so it's worth getting the shape of the idea clear before any of the machinery shows up.

We'll build the smallest honest example of it that runs: a five-line learner that fits a single number to a labeled toy set by minimizing a loss, plus its unsupervised twin that finds groups in the same points with the labels stripped off. Then scikit-learn does both in one line each, using the exact .fit / .predict interface every model in this course wears. No neural networks, no gradients, nothing you can't hold in your head. The point isn't the model — the model here is deliberately dumb. The point is the loop: pick a model family, define a loss, minimize it on data, then check it still works on data you haven't seen. Learn to see that loop and you've learned to read the rest of the book.

A bit of history

The term is older than most of the field thinks. In 1959 Arthur Samuel, an engineer at IBM, wrote a checkers program that got better by playing games against itself and adjusting how it valued board positions, and in the paper describing it he coined the phrase "machine learning" — a computer improving at a task from experience rather than from a programmer spelling out every move. His checkers player eventually beat respectable amateurs, which in the late 1950s was genuinely startling, because the alternative everyone assumed was that you'd have to hand-code every rule of good play yourself.

A year earlier, in 1958, Frank Rosenblatt had built the perceptron — a machine that adjusted a set of weights until it could separate two classes of input, learning the boundary from examples instead of being told where it was. It was the first learning algorithm that looked like what we do now: features in, a weighted decision out, weights nudged by the mistakes it made. The idea got oversold, ran into what a single perceptron provably couldn't do, and the whole approach went quiet for years.

What came back wasn't the perceptron so much as its premise. Through the 1970s and 80s the ambitious way to build an intelligent system was the expert system: sit an engineer next to a specialist and transcribe the specialist's knowledge into thousands of hand-written if-then rules. It worked until it didn't — the rule bases grew brittle, contradicted themselves, and nobody could maintain them. The shift that made modern ML was giving up on writing the rules and going back to Samuel's and Rosenblatt's bet: show the machine enough labeled examples and let it infer the rule. That bet is the whole subject. Everything in this course is a way to make it pay off.

The intuition

Here's the difference between programming and machine learning, in one line. Programming is when you know the rule and write it down. Machine learning is when you don't know the rule, but you have examples of it being followed, and you'd like the computer to recover it.

Say you want to sort incoming email into spam and not-spam. You could try to write the rules — flag anything with "free money," anything from an unknown sender, anything with too many links. You'll be at it forever, the rules will fight each other, and spammers will route around every one you write. Or you can hand a model ten thousand emails people already sorted, and let it find the pattern that separates them. When the rule is short and you know it, write it — don't drag a model into deciding whether a number is even. When the rule is a tangle of ten thousand fuzzy exceptions nobody can fully state, learning it from examples is the only thing that scales.

There are two flavors of "learn it from examples," and the cleanest way to see the split is to look at the same points twice. On the left, every point comes with a label you provided — this one's class 0, that one's class 1 — and the model's job is to learn a boundary that reproduces your labels. That's supervised learning: you supplied the answer key, the model learns to match it. On the right, the same points arrive with no labels at all, and the model's job is to find whatever groups are already sitting in the data on their own. That's unsupervised learning: no answer key, just structure.

Same eighty points, two different questions. On the left the model draws one vertical line — the orange boundary — because the labels you gave split the plane left from right, and that line reproduces your split. On the right, nobody mentioned left or right; the model found four blobs, because four blobs are what the data actually contains. Neither picture is more correct than the other. They're answering different questions of the same points, and which one you're in depends entirely on whether you brought labels.

There's a third flavor this course mostly sets aside: reinforcement learning, where there's no fixed dataset at all. An agent takes actions, gets rewards or penalties, and learns a policy that racks up reward over time — Samuel's self-playing checkers program was an early taste of it. It's a different setup with different math, and it's not what the next thirty-odd chapters are about, so I'll name it and move on.

The math

Strip the vocabulary away and supervised learning is one optimization problem. You have a dataset of $N$ examples, each a feature vector paired with a target:

Stack the feature vectors into a matrix $X$ of shape $N \times d$ (one row per example, one column per feature) and the targets into a vector $y$. You pick a family of candidate rules $f_{\theta}$ indexed by parameters $\theta$ — the model — and a prediction is just the model applied to an input:

A loss $\ell(\hat{y}, y)$ scores how wrong a single prediction is. Learning is choosing the $\theta$ that makes the average loss over the whole dataset as small as it goes:

That single line is the engine of the entire field. Change $f_{\theta}$ and you change the model — a threshold, a line, a tree, a network. Change $\ell$ and you change what "wrong" means — squared error, cross-entropy, hinge loss. Change how you solve the $\arg\min$ and you change the training algorithm — brute force, gradient descent, a closed-form formula. Pick those three and you've specified a learning method completely. That's not a simplification for a first chapter; it's the actual structure, and I'll keep pointing at it all book.

Our concrete model this chapter is the simplest one that isn't a constant: a threshold on a single feature, predicting class 1 when the feature clears it.

And our loss is the plainest one there is — the 0-1 loss, one when the prediction is wrong and zero when it's right. Its average over the data is the misclassification rate, a unitless number between 0 and 1:

Unsupervised learning fits the same template with one term missing: there's no $y_i$, so the loss can only score a prediction against the data itself, not against an answer key. For grouping by nearest center, the parameters are the center locations $\mu_j$, and each point pays the squared distance to whichever center it's assigned to:

Same shape — features in, a rule with parameters, a loss to minimize — just no labels feeding it. That missing $y$ is the whole difference between the two sides of the picture above.

What it's good at, what it isn't

The honest case for machine learning is narrow, and it's worth saying plainly because the field oversells it. Reach for learning when the rule is real but too big or too fuzzy to write by hand — when there genuinely is a pattern separating spam from not-spam, or tumors from healthy tissue, but it lives in a thousand interacting features and no expert can dictate it as clean if-then logic. That's where a model earns its keep: it reads the pattern off the examples that you could never finish transcribing. The other real win is drift. Hand-coded rules rot when the world moves; a model you can retrain on fresh data keeps up.

The case against it is just as real. If you know the rule, write the rule — a model that decides whether a number is positive is slower, dumber, and less reliable than one comparison. Learning needs data, and enough of it that the examples actually cover the pattern; on a few dozen rows you're usually better off with a heuristic. And a learned model is opaque in a way a rule isn't — you get an answer, not a reason, which is a problem the moment someone asks why. The uncomfortable truth underneath all of it is that the algorithm matters less than the data. A mediocre model on clean, representative, well-labeled data beats a clever one on garbage every time, and most of the real work in this job is the data, not the model. Keep that in your pocket for the whole course.

The data

To make "learning = minimizing loss" concrete you need data you can see all of, so it's a seeded toy set: eighty points in two dimensions, arranged as four Gaussian blobs in a two-by-two grid, generated with a fixed random seed so the snapshot never moves. It carries two kinds of truth on purpose. The labels split the points left from right by their x-coordinate — forty class 0 on the left, forty class 1 on the right — which is what the supervised learner will try to reproduce. And the four blobs are a second, independent structure that the unsupervised learner will try to find without ever seeing a label. The blobs are spaced far enough apart that a human sees the four groups instantly, which is exactly what you want when you're checking whether an algorithm agrees with you. That's the same set drawn twice in the picture above.

Build it, one function at a time

Five short functions, no library, and they stack in the order you'd write them: the model first, then the loss that scores it, then the search that minimizes the loss — and then the same shape again with the labels removed.

The model is the decision rule, and it's one comparison. Predict class 1 when the feature clears the threshold:

def predict(x, theta):
    """The model f_theta: predict class 1 when feature x clears the threshold.

    x broadcasts, so this is one point against one theta, or a whole column of
    points against one theta, or a column of points against a row of candidate
    thetas — the same three characters of code either way.
    """
    return (x >= theta).astype(int)

Because that comparison broadcasts, the same three characters handle one point against one threshold or a whole column of points against one threshold — we lean on that in the search. Next, the loss: how often the rule is wrong across the labeled data.

def loss(theta, x, y):
    """Average 0-1 loss: the fraction of labeled points the rule gets wrong.

    Unitless, in [0, 1]. Zero means every prediction matched its label; 0.5 is
    a coin flip. This single number is what 'learning' is going to minimize —
    there is no other objective hiding anywhere.
    """
    return float((predict(x, theta) != y).mean())

This is the number learning minimizes, and there is nothing else. Zero means the rule matched every label; a half is a coin flip. Now the learner itself, and this is the part worth slowing down for, because it's the whole idea in three lines:

def fit(x, y, candidates):
    """Learning, in three lines: score every candidate threshold by its loss on
    the data, then keep the one whose loss is smallest.

    There is no formula that hands you theta. You try, you measure, you choose
    what fits the data best. Returns the winning theta and the loss at every
    candidate (so the chapter can plot the objective coming down).
    """
    losses = np.array([loss(t, x, y) for t in candidates])
    theta_star = float(candidates[int(np.argmin(losses))])
    return theta_star, losses

There's no formula that computes $\theta$ for you. You take a list of candidate thresholds, measure the loss of each one on the data, and keep the candidate whose loss is smallest. Try, measure, choose — that is what "learning from data" means, stripped of every other word. Everything fancier in this book is a cleverer way to do this same search when the candidates are too many to list.

Now the unsupervised twin. With the labels gone the loss can't compare against an answer key, so it compares points against each other: put each point with the center it's nearest to.

def nearest_center(points, centers):
    """Assign each point to the center it's closest to.

    One (N, k) grid of squared distances, argmin down the centers axis. With no
    labels to match, 'nearest center' is the only handle the algorithm has.
    """
    d = ((points[:, None, :] - centers[None, :, :]) ** 2).sum(axis=2)  # (N, k)
    return d.argmin(axis=1)

That's one assignment step. On its own it needs centers to assign to, and good centers are exactly what we don't have, so we alternate: assign points to the nearest center, then move each center to the mean of the points that chose it, and repeat until nothing moves.

def cluster(points, k, seed=0, iters=10):
    """The unsupervised twin of fit: no y, so 'learning' is discovering groups.

    Start from k points chosen at random, then repeat two moves: assign every
    point to its nearest center, and slide each center to the mean of the points
    that chose it. The centers walk into the middle of each blob and stop.
    """
    rng = np.random.default_rng(seed)
    centers = points[rng.choice(len(points), k, replace=False)].astype(float)
    labels = nearest_center(points, centers)
    for _ in range(iters):
        centers = np.array([points[labels == j].mean(axis=0) for j in range(k)])
        new_labels = nearest_center(points, centers)
        if np.array_equal(new_labels, labels):
            break
        labels = new_labels
    return labels, centers

The centers start on random points and walk into the middle of each blob, pulled by the points that keep choosing them. No labels ever enter the function — it finds the four groups from the geometry alone. Same skeleton as the supervised side: parameters, a notion of cost, a loop that drives the cost down. The only thing that changed is that the cost stopped needing $y$.

Watch it work

Here's the supervised learner running, and it's the one animation to sit with in this chapter because it's learning in its entirety with nothing hidden. The top panel is the toy points, colored by their true label, with the orange line the current threshold. A point gets a red ring the instant the rule on its side of the line disagrees with its label. The bottom panel is the loss — the fraction of points currently misclassified — traced as the threshold slides.

Press play. The line starts far to the left, where the rule calls every point class 1 and half of them are wrong, so the loss sits up near a half. As the line sweeps right the red rings clear out, the loss slides down the curve to its minimum, and then as the line keeps going past the gap the rings light up on the other side and the loss climbs back. Learning is that descent to the bottom of the curve, and nothing more. Reset and run it again — it's the same every time, because the search is deterministic.

The curve the bottom panel traces out is the loss as a function of the one parameter we're tuning, and it's worth seeing on its own with the winner marked. The minimum here is a flat valley, not a single point — the labels split cleanly at the gap between the left and right blobs, so any threshold that lands in that gap gets every point right and scores a loss of zero. Our search takes the first one it reaches:

That a perfect rule exists at all is a luxury of clean toy data. Real datasets almost never hand you a zero-loss threshold; the valley bottoms out somewhere above zero, and where it bottoms out is the floor of mistakes the best rule in your family still makes. The shape is the lesson, not the specific zero.

The full implementation

The whole file, top to bottom — both learners, no library. This is exactly what the animation ran:

"""What machine learning is, reduced to code you can hold in your head.

Two tiny learners, no ML library anywhere in this file:

  * a supervised one — given features X and labels y, learn a threshold rule
    f_theta(x) = 1[x >= theta] by picking the theta with the smallest average
    loss on the data. That last clause is the whole definition of learning.

  * an unsupervised one — given the same points with the labels taken away,
    find structure by grouping each point with the center it's nearest to, and
    letting the centers chase the density.

Everything is pure NumPy. Each function shows up in the chapter one step at a
time; the `# region:` markers are what the book's include directives pull in.
"""

import numpy as np


# region: predict
def predict(x, theta):
    """The model f_theta: predict class 1 when feature x clears the threshold.

    x broadcasts, so this is one point against one theta, or a whole column of
    points against one theta, or a column of points against a row of candidate
    thetas — the same three characters of code either way.
    """
    return (x >= theta).astype(int)
# endregion


# region: loss
def loss(theta, x, y):
    """Average 0-1 loss: the fraction of labeled points the rule gets wrong.

    Unitless, in [0, 1]. Zero means every prediction matched its label; 0.5 is
    a coin flip. This single number is what 'learning' is going to minimize —
    there is no other objective hiding anywhere.
    """
    return float((predict(x, theta) != y).mean())
# endregion


# region: fit
def fit(x, y, candidates):
    """Learning, in three lines: score every candidate threshold by its loss on
    the data, then keep the one whose loss is smallest.

    There is no formula that hands you theta. You try, you measure, you choose
    what fits the data best. Returns the winning theta and the loss at every
    candidate (so the chapter can plot the objective coming down).
    """
    losses = np.array([loss(t, x, y) for t in candidates])
    theta_star = float(candidates[int(np.argmin(losses))])
    return theta_star, losses
# endregion


# region: nearest_center
def nearest_center(points, centers):
    """Assign each point to the center it's closest to.

    One (N, k) grid of squared distances, argmin down the centers axis. With no
    labels to match, 'nearest center' is the only handle the algorithm has.
    """
    d = ((points[:, None, :] - centers[None, :, :]) ** 2).sum(axis=2)  # (N, k)
    return d.argmin(axis=1)
# endregion


# region: cluster
def cluster(points, k, seed=0, iters=10):
    """The unsupervised twin of fit: no y, so 'learning' is discovering groups.

    Start from k points chosen at random, then repeat two moves: assign every
    point to its nearest center, and slide each center to the mean of the points
    that chose it. The centers walk into the middle of each blob and stop.
    """
    rng = np.random.default_rng(seed)
    centers = points[rng.choice(len(points), k, replace=False)].astype(float)
    labels = nearest_center(points, centers)
    for _ in range(iters):
        centers = np.array([points[labels == j].mean(axis=0) for j in range(k)])
        new_labels = nearest_center(points, centers)
        if np.array_equal(new_labels, labels):
            break
        labels = new_labels
    return labels, centers
# endregion

The library version

Nobody writes the search by hand in practice, and once you've seen it you don't need to. Every model in scikit-learn — and in every later chapter of this book — wears the same three-move interface: construct it, call .fit(X, y) to learn the parameters, call .predict(X) to use them. A depth-1 decision tree is exactly our one-feature threshold, so the supervised twin is one line of setup and two of calling:

def sklearn_threshold(x, y):
    """The supervised twin of our fit(): a depth-1 decision tree is exactly a
    one-feature threshold rule. Construct, .fit(X, y), .predict(X).

    sklearn expects X as a 2-D array (rows = samples, columns = features), so
    the single feature gets reshaped to a column. Returns the learned threshold
    and the predictions.
    """
    model = DecisionTreeClassifier(max_depth=1, random_state=0)
    model.fit(x.reshape(-1, 1), y)
    theta = float(model.tree_.threshold[0])
    preds = model.predict(x.reshape(-1, 1))
    return model, theta, preds

The difference under the hood is small and worth knowing: our search minimizes raw misclassification, the tree minimizes Gini impurity. Different loss, and on this separable data they land in the same gap, so it doesn't matter here — but that's exactly the "change the loss" knob from the math section showing up in the wild. The unsupervised twin wears almost the same interface, with the one change the whole chapter has been about — there's no y to pass to .fit, and the estimator hands back groups it discovered rather than labels you supplied:

def sklearn_cluster(points, k):
    """The unsupervised twin of our cluster(): scikit-learn's KMeans. Almost the
    same interface — there's no y to pass to .fit, and .fit_predict hands back
    discovered cluster ids instead of known labels. Same Lloyd loop we wrote by
    hand, with k-means++ seeding and restarts done for us.
    """
    model = KMeans(n_clusters=k, n_init=10, random_state=0)
    labels = model.fit_predict(points)
    return model, labels, model.cluster_centers_

Learn this interface once and the rest of the library is free. A random forest, a gradient booster, a support vector machine — you call every one of them exactly like this, which is the quiet reason scikit-learn won.

Scratch versus library

Now the loop the whole course runs on: fit on one slice of data, evaluate on a slice the model never saw. We split the toy set 70/30, learn the threshold on the 56 training points, and score accuracy — the fraction of held-out points classified correctly, a unitless number from 0 to 1 — on the 24 test points, for both our search and sklearn's tree.

Both reach 1.000 test accuracy — every held-out point on the right side of the line. That the two land on the identical number, from two different thresholds (our search stops at the left edge of the gap, sklearn's tree at −0.18), is the point: any rule in the gap is perfect, so the model family and the loss matter, not the particular winner. Don't read a triumphant 1.000 into this, though. It's 1.000 because the toy data is cleanly separable and the rule family happens to contain a perfect answer. That's the exception, not the rule — I engineered the data that way so the machinery would be visible. On real data the number sits somewhere below one, and the gap between the training score and this held-out score is where you find out whether the model learned the pattern or just memorized the training set.

The unsupervised side has no labels to score against during training, so we grade it after the fact against the four blob ids we kept sealed the whole time, using the adjusted Rand index — a unitless agreement score from 0 to 1, where 1 is a perfect match and 0 is chance. Both our from-scratch clustering and sklearn's KMeans score 1.000: every point landed in the group it was really drawn from, recovered from the geometry alone without a single label. Same result, same shape of loop, the answer key opened only at the end to check.

Takeaways

The one thing to carry out of this chapter is the loop, because every chapter that follows is a variation on it: pick a model family, define a loss that says what a good fit means, minimize that loss on your data, then check the result on data you held out. Threshold, tree, forest, network — the model changes, the loss changes, the way you solve the minimization changes, but the loop doesn't. Once you can look at any method and name its three pieces, the book stops being a list of unrelated algorithms and becomes one idea in thirty costumes.

The engineer's version of that idea is a decision rule about when to even reach for it. Machine learning is worth the trouble when the rule you want is real but too big or too fuzzy to write by hand, and it moves often enough that hand-written rules would rot. When the rule is short and stable, write the rule — you'll ship faster and sleep better. And when you do reach for a model, spend your time on the data before the algorithm, because clean, representative, honestly-labeled data beats a clever model on messy data nearly every time, and the reverse almost never happens. The rest of this course teaches you a shelf of model families and losses to choose among. This chapter is the reason any of them work: they're all just ways of fitting a function to data, and then being honest about whether it generalizes.