Gaussian mixture models

k-means, with two assumptions relaxed.

Clusters get to be tilted ellipses, not circles. Points get to belong to more than one at once.

Fit by EM — the same soft-assign-then-refit loop, now in 2-D with a full covariance per component.

Two upgrades over k-means

A cluster is a full Gaussian: a mean (center) and a covariance (shape — width, height, and tilt). Full covariance = ellipse at any angle.
Assignment is soft: each point gets a probability of belonging to each component. Deep inside → near-certain. On a seam → a split.

Those probabilities are the responsibilities. They're how it fits and what it hands back.

The data

300 points, three blobs, sheared into long diagonal overlapping ellipses. No labels. The canonical case k-means gets wrong.

The loop (EM)

E-step — responsibility of component k for point i:

\gamma_{ik} = \frac{\pi_k \, \mathcal{N}(x_i \mid \mu_k, \Sigma_k)}{\sum_{j} \pi_j \, \mathcal{N}(x_i \mid \mu_j, \Sigma_j)}

M-step — re-fit each component, weighted by responsibility. The covariance is what tilts the ellipse:

\Sigma_k = \frac{1}{N_k} \sum_{i=1}^{N} \gamma_{ik}\, (x_i - \mu_k)(x_i - \mu_k)^{\top}

The log-likelihood only climbs. When it flattens, stop.

Watch the ellipses mold

Two contours per component (1σ, 2σ) from the real covariance eigen-decomposition. Points shaded by soft responsibility — blends at the seams. Lower panel: the log-likelihood rising.

Round seeds → diagonal ellipses in three or four frames. Converges at 23 iterations, log-likelihood −1,081 → −515.

Picking k: BIC

Log-likelihood always improves with more components. BIC penalizes the parameter count, so it has a real minimum.

Minimum at k=3. The elbow question with a real objective under it.

The money shot

Same data. k-means cuts a straight line across the diagonal groups. GMM wraps each in its tilted ellipse.

Adjusted Rand index vs the true blobs: k-means 0.60, GMM 1.0.

Scratch vs library

From-scratch EM and sklearn.mixture.GaussianMixture(covariance_type="full") both hit average log-likelihood −1.7162, labels agree at ARI 1.0.

Takeaways

Reach for a GMM when k-means is the right instinct but the wrong shape: blobs around centers, but stretched, tilted, overlapping.
Full covariance → shaped clusters. Soft responsibilities → honesty at seams. It's a density model → score new points, per-point confidence, BIC for k.
Costs: many parameters (overfits in high-D, covariances can collapse — keep the ridge), and more local optima than k-means (seed well, restart).
Same E/M/log-likelihood pattern runs HMMs, topic models, half of unsupervised learning. Build the mixture once and you've built the pattern.