Topic models

You have a pile of documents and no labels. EM discovers the themes — which words hang together, which document is about what — without ever being told.

A mixture of multinomials fit by the same E-step/M-step engine as Gaussian mixtures. This is pLSA, the ancestor of LDA.

The generative story

Each document is written by ONE latent topic.
A topic is a multinomial over the vocabulary — a bias toward certain words.
To write a document: pick a topic from the prior, then draw its bag of words from that topic's word distribution.

We see only the word counts. We know neither the topics nor which one wrote each document. So we alternate.

The math

Generative story for document $i$ with word counts $x_i$ :

p(x_i \mid \theta) = \sum_{k=1}^{K} \pi_k \prod_{w=1}^{V} \phi_{kw}^{\,x_{iw}}

E-step — posterior that topic $k$ wrote document $i$ :

W_{ik} = \frac{\pi_k \prod_w \phi_{kw}^{\,x_{iw}}}{\sum_j \pi_j \prod_w \phi_{jw}^{\,x_{iw}}}

M-step — responsibility-weighted word counts, normalized per topic:

\phi_{kw} = \frac{\sum_i W_{ik}\, x_{iw}}{\sum_i W_{ik} \sum_{w'} x_{iw'}}

Watch the topics emerge from noise

Three panels, one per topic. Every panel starts flat — all words at the chance rate — and sharpens onto its own five planted words. Bar color = the topic that truly planted each word.

The guarantee holds here too

Same monotone climb as every EM. The data log-likelihood rises from −16,365 at the random start to −11,648 at convergence, and never drops.

Scratch vs. library

From-scratch mixture-of-multinomials EM recovers the planted topics: document adjusted Rand index $1.00$ , mean topic-word cosine $0.998$ .
sklearn LatentDirichletAllocation (variational LDA — a different, richer model): ARI $1.00$ , cosine $0.998$ .
Both nail this clean corpus. LDA is the fully Bayesian descendant of the model we built.

Takeaways

Topic models are EM for text: swap the Gaussians for multinomials over words.
pLSA (Hofmann, 1999) — one topic per document, fit by EM. LDA (Blei, Ng & Jordan, 2003) — a Dirichlet prior over topic mixtures, fit by Bayesian inference.
Still find them in short-text and interpretable-clustering work; embeddings and LLMs took over open-ended "what is this about."
The engine is the same E-step/M-step you already know.