Principal component analysis

Find the axes along which the data varies most. Keep the top few. Drop the rest.

Unsupervised dimensionality reduction — no labels, just structure.

The idea

A tilted, correlated cloud is redundant: one long direction carries almost everything, the perpendicular one carries the leftovers.

PC1 points along the greatest variance. PC2 is perpendicular, holding the rest. New axes, mixtures of the old, chosen so the first captures the most spread.

Keep PC1, drop PC2, and the picture barely changes.

The data

150 points, a Gaussian blob stretched and tilted about 33 degrees off axis.

The math

Center, then covariance, then its eigenvectors:

\tilde{x}_i = x_i - \bar{x}

C = \frac{1}{N-1} \tilde{X}^{\top} \tilde{X}

C v_j = \lambda_j v_j

Each eigenvector is a principal direction; its eigenvalue is the variance along it. Project onto the top k: $z_i = V_k^{\top} \tilde{x}_i$ .

Watch it work

Axes appear. Cloud rotates so PC1 lies flat. Every point drops onto PC1; the gray stubs are the discarded PC2 residuals. Real eigenvectors, real change of basis.

PC1 alone holds 92.7% of the variance, so the drop barely moves anything.

Iris: four dimensions to two

PCA sees four numbers per flower, never the species. Two components carry 97.77% of the variance — the plot below is almost the whole dataset.

The species fall out of the projection without ever being shown.

Scratch vs. library

Covariance eigendecomposition vs. sklearn's SVD. Same answer: PC1 0.9246, PC2 0.0531. Components agree to ~1e-14 after correcting for the sign ambiguity.

An eigenvector is defined up to sign — a flipped component is the ambiguity, not a bug.

Takeaways

Compress, visualize, decorrelate, denoise — one eigendecomposition, four jobs.
No hyperparameters, no seed, no iteration. Cheap and deterministic.
It's linear: variance and pairwise correlation only. It flattens curves and manifolds.
Variance is not always importance. The loudest direction isn't always the useful one.
Next: t-SNE, UMAP, kernel PCA, autoencoders — PCA with the straight-line constraint relaxed.