Gradient boosting

Guess the mean. Fit a tree to what's left over. Add a shrunken slice. Repeat.

AdaBoost's idea generalized: fit the gradient of the loss. For squared error, the gradient is just the residual.

Forest vs. boosting

Random forest: trees in parallel, averaged — independent guesses.
Boosting: trees in sequence, each one fixing the last one's mistakes.
The "mistake" it fits is the residual y - F(x) — the leftover.
Weak learners: deliberately shallow trees (depth 2–3), one small nudge each.

The data

A 1-D curve, y = sin(x) + 0.25x + noise, 120 points. A sum of shallow trees is a staircase — watch it build toward the curve.

The math

Additive model, built one tree at a time:

F_M(x) = F_0 + \nu \sum_{m=1}^{M} h_m(x)

Squared-error loss, and its negative gradient at the current prediction:

L(y, F) = \tfrac{1}{2}\,(y - F)^2

r_i = -\left.\frac{\partial L(y_i, F)}{\partial F}\right|_{F = F_{m-1}(x_i)} = y_i - F_{m-1}(x_i)

The negative gradient is the residual. Fit a tree to it, take a shrunken step:

F_m(x) = F_{m-1}(x) + \nu\, h_m(x)

Watch it build

One boosting round per frame, learning rate 0.1. Top: F(x) creeping toward the curve. Bottom: the residuals collapsing. Train MSE 1.007 → 0.069 over 50 rounds.

The learning rate is the knob

Test MSE vs. trees on real data, three learning rates. Small rate: lower floor, more trees, forgiving. Big rate: fast then overfits off a cliff.

Scratch vs. library

100 trees, lr 0.1, depth 3 on diabetes. Scratch R² 0.3766, sklearn R² 0.3916 — neck and neck. A single depth-3 tree scores 0.2564; boosting a hundred of them buys the climb to 0.38.

Takeaways

Gradient boosting = fit the negative gradient of the loss with a tree, step, repeat.
Squared error → the gradient is the residual. Guess mean, fit leftover, shrink, add.
The tabular-data champion: it beats a forest by correcting errors, not averaging.
Learning rate × tree count trade off: small rate, many trees, early stopping.
Sharpen this skeleton — better splits, regularization, speed — and you get XGBoost. Next.