The EM algorithm

k-means makes a hard choice for every point. EM refuses to guess — it hands each point a probability of belonging to each group.

Expectation-Maximization: the load-bearing recipe for latent-variable and missing-data problems.

The chicken and egg

If you knew which Gaussian made each point, fitting the Gaussians is a one-liner.
If you knew the Gaussians, guessing the labels is a one-liner.
You know neither. So alternate:
- E-step — guess the labels softly from the current Gaussians (responsibilities).
- M-step — refit the Gaussians from those soft labels.
Repeat until the numbers stop moving.

E-step — responsibility of component $k$ for point $i$ :

\gamma_{ik} = \frac{\pi_k\, \mathcal{N}(x_i \mid \mu_k, \sigma_k^2)}{\sum_j \pi_j\, \mathcal{N}(x_i \mid \mu_j, \sigma_j^2)}

M-step — responsibility-weighted updates:

\pi_k = \frac{1}{N}\sum_i \gamma_{ik}, \quad \mu_k = \frac{\sum_i \gamma_{ik} x_i}{\sum_i \gamma_{ik}}

Two bells fighting over a column of numbers. Points colored by soft responsibility; the log-likelihood climbs — and never drops.

EM's central promise: the log-likelihood never decreases. Not "usually" — never, every iteration, provably.

That monotonic climb is the whole reason the method is trusted.

Fit from a deliberately bad start; land on the same answer as sklearn.mixture.GaussianMixture.

EM is the go-to for latent-variable and missing-data problems.
It only finds a local optimum — initialization matters; restart and keep the best.
It underlies Gaussian mixtures, topic models, and HMMs.
Next: the full 2-D story with tilted covariance ellipses — Gaussian mixture models.