Linear regression

Fit a line — a plane, really — through the data and read predictions off it.

The workhorse baseline for regression: readable coefficients, exact solution, the model everything fancier has to beat.

Two ways to the same line

Normal equation: $w = (X^{\top}X)^{-1} X^{\top} y$ — one linear solve, exact.
Gradient descent: step against $\nabla_w L = \frac{2}{n} X^{\top}(Xw + b - y)$ until it stops moving.
The loss is a convex bowl, so there's one minimum and every correct method finds it.
The closed form is what you ship; the gradient loop is what trains everything later.

Least squares: pick the line where the squared vertical gaps add up to the smallest total.

Every frame is one gradient step. The line rotates into place; the MSE falls off a cliff, then flattens.

After 60 steps, gradient descent matches the closed form: slope 3.00, intercept 5.10.

The model, per row and stacked:

\hat{y}_i = w^{\top} x_i + b, \qquad \hat{y} = Xw + b

The thing we minimize — mean squared error:

L(w, b) = \frac{1}{n} \sum_{i} \left( \hat{y}_i - y_i \right)^2

A convex bowl. Solve it directly, or descend it. Same bottom.

Normal equation, gradient descent, and sklearn on California housing — identical.

Test R² 0.585, MSE 0.528 — the same for all three. There's only one least-squares line.

The fan shape is heteroscedasticity; the diagonal streak is the $500k price cap.

Fit the line first. R² is the bar; the coefficients are a free sanity check.
Median income coefficient ≈ 0.44 → +$10k income predicts +$43,900 in value.
Watch the signs: correlated features (rooms vs. bedrooms) make coefficients unstable.
The normal equation is the last closed form you get. Gradient descent is what carries the rest of the course.
Keep the residual plot — it's where linearity and constant variance visibly break.