AdaBoost

Week 1's decision stump was the weakest useful model there is.

This chapter stacks fifty of them into one of the strongest classifiers of its era.

Each new stump is aimed at the last one's mistakes.

The whole algorithm

Carry a weight on every training point. Start them uniform, then repeat:

Fit the stump with the smallest WEIGHTED error.
Give it a vote α sized to how good it was.
Raise the weight on the points it missed, lower it on the hits.

The final call is a weighted vote of every stump, then take the sign.

The data

200 points, two concentric classes from make_gaussian_quantiles. Inner blob vs surrounding shell.

No axis-aligned line splits this — a single stump tops out near 0.67.

The math

Weighted error, then the vote:

\varepsilon_t = \sum_{i=1}^{N} w_i^{(t)} \, \mathbb{1}\!\left[\, h_t(x_i) \neq y_i \,\right]

\alpha_t = \tfrac{1}{2} \ln\!\left( \frac{1 - \varepsilon_t}{\varepsilon_t} \right)

Reweight — misses grow, hits shrink — then vote by sign:

w_i^{(t+1)} = \frac{w_i^{(t)} \, \exp\!\left( -\alpha_t \, y_i \, h_t(x_i) \right)}{Z_t}

H(x) = \operatorname{sign}\!\left( \sum_{t=1}^{T} \alpha_t \, h_t(x) \right)

Watch it work

Point SIZE is its current weight. Missed points swell and pull the next stump. Orange line: the new stump. Shaded grid: the ensemble boundary sharpening.

Every stump stays terrible (ε ≈ 0.4). The intelligence is in the accumulation.

More rounds isn't free

Train error slides to zero. Test error bottoms near round 17, then drifts up — later rounds chase the noise in the class overlap.

Scratch vs. library

Single stump 0.63. Our AdaBoost 0.88. sklearn's 0.92. Both boosted models hit 0.98 on training.

Takeaways

A strong model can be a swarm of dumb ones, aimed. That's boosting.
Almost no knobs: the weak learner and the number of rounds.
It chases hard points, so it can't tell a hard case from a mislabeled one — noisy data makes it overfit. Clean data only.
Next: gradient boosting fits each learner to the residual instead of reweighting points — the same move, generalized to any loss. The road to XGBoost.