Learning without labels

What this chapter covers

Everything up to here had an answer key. There was a label column, a target to match, and a number — accuracy, RMSE, R² — that told you flatly whether the model was right. This chapter takes the answer key away and doesn't give it back. From here on the data is just points, no truth attached, and the job changes from predicting an answer to finding the structure that's already sitting in the data.

That's unsupervised learning, and it's a different job with a different failure mode. When there's no label, there's no accuracy, which means the hard part stops being "can I fit it" and becomes "how do I know it worked at all." This is the chapter that names the families you're about to meet — clustering, dimensionality reduction, density estimation, anomaly detection — and then spends most of its time on the honest question none of them answer for free: how you judge a result when nothing tells you the right one. We build the one metric that judges a clustering with no labels, the silhouette score, by hand in NumPy, and use it to do the thing you can never do on real unlabeled data — watch a quality number peak at exactly the structure that was really there.

The four chapters after this one are the methods themselves. Principal component analysis compresses. K-means, hierarchical clustering, and Gaussian mixtures group. This chapter is the frame around all four: what the whole enterprise is for, and how you tell a good answer from a confident wrong one.

A bit of history

The idea of finding groups without being told the groups is older than machine learning and came out of the sciences that had piles of measurements and no theory to sort them. In 1939 the psychologist Robert Tryon published Cluster Analysis, a book-length method for grouping variables by how they correlated, worked out by hand for a discipline drowning in questionnaire data. Around the same time anthropologists Harold Driver and Alfred Kroeber were doing much the same thing to culture traits, computing similarities between tribes and grouping them — the phrase "cluster analysis" is theirs and Tryon's, decades before a computer ran one.

The other root is factor analysis, and it's older still. In 1904 Charles Spearman noticed that childrens' scores across unrelated school subjects all correlated, and posited a single hidden factor — general intelligence — driving them, which he backed out of the correlation matrix. Louis Thurstone generalized it to multiple factors in the 1930s. Strip the psychology and factor analysis is dimensionality reduction: many observed variables explained by a few latent ones, which is the exact idea that becomes principal component analysis in the next chapter. The whole unsupervised tradition grew from this one impulse — there's more structure in the data than the columns admit, and you can recover it without anyone labeling it first.

The intuition

Unsupervised learning isn't one method, it's a handful of jobs that share the same handicap. It helps to hold the map in your head before the individual chapters, because they're all answers to "the data has no labels — now what."

Clustering groups similar points: k-means, hierarchical clustering, Gaussian mixtures. You believe the data falls into kinds and you want the machine to find the kinds. Dimensionality reduction compresses: PCA takes fifty correlated columns and hands you two axes that keep most of the variation, so you can plot what you couldn't before or feed a smaller table to a supervised model. Density estimation learns where the data lives — the shape of the distribution itself — which turns directly into anomaly detection: a point in a low-density nowhere is a candidate outlier, a fraud, a broken sensor. Different jobs, one thing in common: no target column, so no accuracy.

Here's the whole difficulty in one picture. On the left is the raw data this chapter works on — a cloud of points, gray, no labels, exactly what the algorithm sees. On the right is the same cloud after a clustering, colored by the groups it discovered. No labels went in. Structure came out.

Your eye did the same thing the algorithm did — it found four groups in the gray version before you read the colored one. The trouble starts when you ask the obvious follow-up: are there really four? Could be three, could be six, and the data won't tell you. That question is the whole chapter, and the tool for answering it is a metric that grades a clustering using only the geometry, no labels required.

The math

The metric is the silhouette, and it scores one point at a time by asking whether that point sits more comfortably in its own cluster than in the nearest rival. Take a point and call the mean distance from it to the other points in its own cluster $a$ — how tight its home is. Then look at every other cluster, average the distance from the point to all the members of each, and take the smallest of those averages; call it $b$ — how far the nearest neighbouring cluster is. The silhouette coefficient of that point is:

Read the two extremes off the formula. If the point's own cluster is much tighter than the nearest other one, $a \ll b$, the fraction goes to $b/b = 1$ — a perfect fit. If the point sits right on the border between two clusters, $a \approx b$ and $s \approx 0$. If it's actually closer to a neighbouring cluster than its own, $a > b$ and $s$ goes negative — the point is likely misassigned. So every point gets a score in $[-1, 1]$, unitless, and the silhouette score of the whole clustering is just the average over all points:

The one edge case is a cluster with a single point in it: there are no other members to average a distance to, so $a$ is undefined. By convention that point scores $s = 0$, which is the same thing scikit-learn does. Everything about the silhouette is distances between points you already have — no target, no truth. That's what makes it usable on the one kind of data where you have no answer key, which is all unsupervised data.

What it's good at, what it isn't

What's genuinely good here is that unsupervised learning works on the data you actually have. Labels are expensive — someone has to sit and tag examples — and the overwhelming majority of data arrives with none. Clustering, compression, and anomaly detection all run on raw unlabeled tables, which is why this family is where most real exploratory work starts. It's how you find the customer segments nobody defined, squeeze a wide feature matrix down to something you can plot, or flag the transaction that doesn't look like the others. It shines for exploration, for compression, and as a feature step feeding a supervised model downstream.

What's hard is the thing that makes it powerful. With no label there is no single number that means "correct," so evaluation splits into two honest but partial options. Internal metrics judge a clustering by its own geometry and need no truth: the silhouette (unitless, in $[-1, 1]$), or inertia, the within-cluster sum of squared distances (data units²). They tell you whether the groups are tight and separated, not whether they mean anything. External metrics like the adjusted Rand index (unitless, in $[-0.5, 1]$, where ≈0 is a random clustering and 1 is perfect agreement) compare your clustering to known labels — but if you had the labels you probably wouldn't be clustering. The trap is treating a good internal score as proof. A clean silhouette says the groups are geometrically crisp; it says nothing about whether those groups are the segments your business cares about. The metric is necessary and nowhere near sufficient, and forgetting that is the classic way unsupervised work goes wrong.

The data

Four Gaussian blobs from scikit-learn's make_blobs, 180 points in 2-D, with a fixed random seed so the snapshot is stable. Two features so the whole thing fits on a page and you can see every point; four centers spaced far enough apart that a human finds the groups instantly, which is exactly what you want when the point is to check whether an unsupervised method agrees with the obvious answer. The generator also returns a true blob id for each point. We commit those, but treat them as a sealed envelope — the coordinates go into the clustering, and the ids come out only at the very end to compute an external metric, so we can grade a result the way you almost never get to on real data.

Build it, one function at a time

The from-scratch code isn't a clustering algorithm this time — it's the ruler. Five short functions that turn a set of points and a clustering into a silhouette score, with no labels anywhere. The clustering itself we borrow from the library; judging it is the part worth building by hand.

Everything rests on distances between points, so we compute them all once. Given the points, we want the full point-to-point distance matrix — every $i$ to every $j$ — and then index into it:

def pairwise_distances(X):
    """Euclidean distance between every pair of points.

    X is (N, D); the result is (N, N), where entry (i, j) is ||x_i - x_j||.
    Broadcasting builds the whole matrix at once — the silhouette needs every
    point-to-point distance, so we pay for it once and index into it after.
    """
    diff = X[:, None, :] - X[None, :, :]        # (N, N, D)
    return np.sqrt((diff ** 2).sum(axis=2))     # (N, N)

That (N, N) matrix is the whole cost of the method; the rest is bookkeeping on top of it. The first quantity is $a$, a point's mean distance to the other members of its own cluster — how tight its home is:

def intra_cluster(D, labels, i):
    """a_i: the mean distance from point i to the OTHER points in its cluster.

    Distances to points in a different cluster are ignored, and the point's
    distance to itself (a zero on the diagonal) is excluded from the average.
    A cluster of size one has no such neighbours, so a_i is undefined and we
    return NaN — the caller turns that into a silhouette of 0.
    """
    same = (labels == labels[i])
    same[i] = False                     # drop the point itself
    n = same.sum()
    if n == 0:
        return np.nan                   # singleton cluster
    return D[i, same].mean()

Note the two exclusions: we drop the point's own row so it isn't measuring distance to itself, and if the cluster has only that one point there's nothing to average, so we return NaN and let the caller score it zero. Next is $b$, the distance to the nearest rival cluster — for each other cluster, the mean distance to its members, then the smallest of those:

def nearest_cluster(D, labels, i):
    """b_i: the smallest mean distance from point i to any OTHER cluster.

    For each cluster that isn't i's own, average i's distance to all its
    points; b_i is the minimum of those averages — the nearest rival cluster,
    the one i would defect to if it left home.
    """
    own = labels[i]
    best = np.inf
    for c in np.unique(labels):
        if c == own:
            continue
        mean_d = D[i, labels == c].mean()
        best = min(best, mean_d)
    return best

With $a$ and $b$ in hand, the per-point silhouette is the formula straight off the page. We walk every point, handle the singleton case as a zero, and combine into $s = (b - a) / \max(a, b)$:

def silhouette_samples(X, labels):
    """The silhouette coefficient s_i for every point.

    Build the distance matrix once, then for each point combine its own-cluster
    tightness a and its nearest-rival distance b into s = (b - a) / max(a, b).
    Points in a singleton cluster score 0 by convention. Returns an (N,) array
    in [-1, 1].
    """
    D = pairwise_distances(X)
    labels = np.asarray(labels)
    s = np.zeros(len(X))
    for i in range(len(X)):
        a = intra_cluster(D, labels, i)
        if np.isnan(a):
            s[i] = 0.0                  # singleton: no self-comparison possible
            continue
        b = nearest_cluster(D, labels, i)
        s[i] = (b - a) / max(a, b)
    return s

That returns one score per point, each in $[-1, 1]$. The score of the whole clustering is the last step, and it's just the mean:

def silhouette_score(X, labels):
    """The silhouette score of a clustering: the mean of s over all points.

    A single unitless number in [-1, 1]. Higher is better — points sit closer
    to their own cluster than to the nearest other. This is the whole label-free
    verdict on how well the clustering carved the data.
    """
    return float(silhouette_samples(X, labels).mean())

One unitless number that says how well the clustering carved the data, computed from nothing but the points and the group assignments. That's the label-free verdict we're going to sweep across k.

Watch it work

Here's the payoff, and the thing you can never do in the wild. We don't know how many clusters the data has — pretend we don't, anyway — so we try every k from 2 to 8. For each one we cluster with k-means and score the result with the silhouette we just built. Press play and watch two things at once: the top panel recolors the points into k groups, and the bottom panel plots the silhouette score for each k as it's computed. The caption names the k and the score.

Watch the bottom curve climb, peak, and fall. At k=2 the silhouette is 0.597 (unitless, in $[-1, 1]$) — the clustering is fusing real groups together. It rises to 0.754 at k=3, then to its peak of 0.788 at k=4, and from there it drops: 0.654 at k=5, 0.566 at k=6, down into the 0.44s by k=8, where k-means is slicing the real blobs into arbitrary pieces. The metric peaks at four, and four is how many blobs we built. Nothing in that sweep saw a single label. The silhouette found the true number of groups from geometry alone, which is the whole promise of internal validation — a label-free number that bends where the real structure is.

The same sweep as a pair of static curves, so you can read the bend directly. The top curve is the silhouette (unitless, in $[-1, 1]$, higher is better, peak marked in orange); the bottom is inertia (data units², the within-cluster sum of squared distances, always falling as k grows, so you read its elbow):

Both curves point at four. The silhouette has an outright peak there — 0.788, the highest score in the sweep. Inertia doesn't peak, it only ever falls, from 4,346 at k=2 to 970.9 at k=3 to 286.6 at k=4 (data units² throughout), and then the fall flattens: k=5 only buys you down to 254.2. That flattening is the elbow, and it sits at the same k the silhouette chose. Two different internal metrics, computed from geometry alone, agreeing on the structure. When they agree this cleanly you can trust it. In the wild they often disagree, and then you're back to judgment.

The full implementation

The whole silhouette, no library, top to bottom. This is exactly what the sweep above scored every clustering with:

"""The silhouette score, built from scratch.

This is a methodology chapter, not an algorithm chapter, so the from-scratch
piece isn't a clustering method — it's the ruler you use to judge one. The
silhouette score asks, for every point, a purely geometric question: does this
point sit closer to its own cluster than to the nearest other cluster? It needs
no labels and no ground truth, only the points and a clustering someone already
produced. Pure NumPy — no ML library in this file.

For point i, with a = its mean distance to the other points in its own cluster
and b = the mean distance to the points of the nearest neighbouring cluster:

    s_i = (b - a) / max(a, b)         in [-1, 1]

s near 1 means the point fits its cluster far better than any other; s near 0
means it sits on a boundary; s below 0 means it would be happier somewhere else.
The silhouette score of a whole clustering is the mean of s over all points.
Singleton clusters have no intra-cluster distance to average, so their s is
defined to be 0 — the same convention scikit-learn uses.

Every function below appears in the chapter one step at a time (the
`# region:` markers are what the book's include directives pull in).
"""

import numpy as np
import pandas as pd


# region: pairwise
def pairwise_distances(X):
    """Euclidean distance between every pair of points.

    X is (N, D); the result is (N, N), where entry (i, j) is ||x_i - x_j||.
    Broadcasting builds the whole matrix at once — the silhouette needs every
    point-to-point distance, so we pay for it once and index into it after.
    """
    diff = X[:, None, :] - X[None, :, :]        # (N, N, D)
    return np.sqrt((diff ** 2).sum(axis=2))     # (N, N)
# endregion


# region: intra
def intra_cluster(D, labels, i):
    """a_i: the mean distance from point i to the OTHER points in its cluster.

    Distances to points in a different cluster are ignored, and the point's
    distance to itself (a zero on the diagonal) is excluded from the average.
    A cluster of size one has no such neighbours, so a_i is undefined and we
    return NaN — the caller turns that into a silhouette of 0.
    """
    same = (labels == labels[i])
    same[i] = False                     # drop the point itself
    n = same.sum()
    if n == 0:
        return np.nan                   # singleton cluster
    return D[i, same].mean()
# endregion


# region: nearest
def nearest_cluster(D, labels, i):
    """b_i: the smallest mean distance from point i to any OTHER cluster.

    For each cluster that isn't i's own, average i's distance to all its
    points; b_i is the minimum of those averages — the nearest rival cluster,
    the one i would defect to if it left home.
    """
    own = labels[i]
    best = np.inf
    for c in np.unique(labels):
        if c == own:
            continue
        mean_d = D[i, labels == c].mean()
        best = min(best, mean_d)
    return best
# endregion


# region: samples
def silhouette_samples(X, labels):
    """The silhouette coefficient s_i for every point.

    Build the distance matrix once, then for each point combine its own-cluster
    tightness a and its nearest-rival distance b into s = (b - a) / max(a, b).
    Points in a singleton cluster score 0 by convention. Returns an (N,) array
    in [-1, 1].
    """
    D = pairwise_distances(X)
    labels = np.asarray(labels)
    s = np.zeros(len(X))
    for i in range(len(X)):
        a = intra_cluster(D, labels, i)
        if np.isnan(a):
            s[i] = 0.0                  # singleton: no self-comparison possible
            continue
        b = nearest_cluster(D, labels, i)
        s[i] = (b - a) / max(a, b)
    return s
# endregion


# region: score
def silhouette_score(X, labels):
    """The silhouette score of a clustering: the mean of s over all points.

    A single unitless number in [-1, 1]. Higher is better — points sit closer
    to their own cluster than to the nearest other. This is the whole label-free
    verdict on how well the clustering carved the data.
    """
    return float(silhouette_samples(X, labels).mean())
# endregion


def load_data(path="../data/blobs.csv"):
    """make_blobs snapshot: 180 2-D points (x, y) plus the true blob id.

    The blob column is ground truth we NEVER cluster on — it exists only so we
    can compute an external metric (the adjusted Rand index) at the very end,
    the honest check you almost never get on real unlabeled data.
    """
    return pd.read_csv(path)

The library version

You wouldn't hand-roll this in production, and once you've built it you don't need to. sklearn.metrics.silhouette_score computes the identical geometry in a C loop:

def sklearn_silhouette(X, labels):
    """The library silhouette: mean s over all points, unitless in [-1, 1].

    Same objective our from-scratch version computes, so the two numbers should
    agree to within floating-point noise on identical inputs.
    """
    return float(sk_silhouette(X, labels))

The clusterings the sweep judged came from sklearn.cluster.KMeans, the tool the next-but-one chapter builds by hand — here it's a two-line call returning labels and inertia:

def cluster(X, k, seed=0):
    """Cluster X into k groups with KMeans. Returns labels and inertia.

    Inertia is the within-cluster sum of squared distances (data units^2) —
    an INTERNAL metric that always falls as k grows, so you read its elbow, not
    its minimum.
    """
    km = KMeans(n_clusters=k, n_init=10, random_state=seed).fit(X)
    return km.labels_, float(km.inertia_)

Those two — a clustering tool and an internal metric — are the whole loop for unsupervised evaluation without labels. The dimensionality-reduction side of the family is sklearn.decomposition.PCA, the next chapter's subject, referenced here as the one-line call it is: many columns in, a few informative axes out:

def pca_project(X, n_components=2, seed=0):
    """Compress X to n_components with PCA — the dimensionality-reduction tool.

    Returns the projected coordinates. On already-2-D data this is a rotation,
    but it's the same one-line call you make to squeeze a 50-column table down
    to two axes you can actually plot. Its own chapter builds it from scratch.
    """
    return PCA(n_components=n_components, random_state=seed).fit_transform(X)

And on the rare day you do have labels to check against, the external metric is adjusted_rand_score, which compares two labelings after correcting for chance:

def sklearn_ari(true_labels, pred_labels):
    """Adjusted Rand index: agreement between two labelings, chance-corrected.

    Unitless; ~0 for a random clustering, 1 for a perfect match, and it can go
    slightly negative for worse-than-random. This is an EXTERNAL metric — it
    needs the true labels, which unsupervised learning almost never has.
    """
    return float(adjusted_rand_score(true_labels, pred_labels))

Scratch versus library

Two things to check. First, that the hand-written silhouette is the real thing: on the chosen k=4 clustering our from-scratch score is 0.7876 and sklearn's is 0.7876 — identical to every digit, silhouette (unitless, in $[-1, 1]$). Same geometry, same answer, so the number the animation peaked on is trustworthy.

Second, the point of the whole chapter: does a good internal score line up with actually recovering the structure? Here we open the sealed envelope and grade the clustering against the true blob ids with the adjusted Rand index. This compares the chosen k=4 against a plausible-but-wrong k=6, on both an internal metric (silhouette) and an external one (ARI):

They agree, and that's the reassuring case. At the chosen k=4 the silhouette is 0.7876 (unitless, $[-1, 1]$) and the adjusted Rand index against the true blobs is 1.0 (unitless, $[-0.5, 1]$) — a perfect recovery, every point in the group it was really drawn from. At the wrong k=6 the silhouette drops to 0.5657 and the ARI to 0.8329: k-means split real blobs to invent two extra clusters, the geometry got worse, and the agreement with the truth got worse in step. The label-free metric and the label-based metric moved together, which is the outcome you hope for and the reason the silhouette is worth trusting as a proxy when the labels aren't there.

But notice what had to be true for that last paragraph to exist: we had labels. The ARI column only happens because this dataset came with a sealed envelope. On the real unlabeled data you'll actually cluster, the right half of that chart doesn't exist. You get the silhouette and the elbow, and you get to decide.

Takeaways

This is the hinge of the course. Everything before it was supervised — a label, a target, an accuracy — and everything after it is not. The mental shift is bigger than the math: you stop asking "is the model right" and start asking "is the result useful," and no number answers the second question for you. Unsupervised learning is exploratory by nature. You run it to see what's in the data, not to hit a score.

Which makes validation the whole job, and the honest version of it is: use internal metrics and your own eyes, never just a number. The silhouette and the elbow are real signal — when they agree, like they did here, believe them — but a crisp silhouette only tells you the groups are geometrically tight, not that they mean anything to the problem. A clustering can score beautifully and be useless, and the only thing that catches that is a domain expert looking at the groups and saying "yes, those are real segments" or "no, that's an artifact." Treat the metric as a filter, not a verdict.

Reach for this family when you have data and no labels, which is most of the time. It's where exploratory analysis starts, it's how you compress a wide table into something you can see or feed forward, it's how you flag the anomaly that doesn't fit, and it's a quiet source of features for the supervised models that come later — cluster membership is just another column. The next four chapters are the methods, one at a time. PCA compresses. K-means, hierarchical clustering, and Gaussian mixtures group, each relaxing an assumption the last one made. Every one of them will hand you an answer with total confidence and no way to check it built in. The silhouette you just built by hand is how you check.