Judging a regressor
Six ways to squeeze predictions-vs-truth into one number: MSE, RMSE, MAE, R²,
adjusted R², MAPE.
They are not interchangeable. Different units, different things rewarded, one of
them lies when an outlier shows up.
The running question: what unit is this number in?
The units, up front
- MSE — target units squared — ($100k)². An area. Don't try to read it.
- RMSE — target units — $100k. The MSE made interpretable.
- MAE — target units — $100k. The robust one.
- R² — unitless, fraction of variance explained, ≤ 1.
- adjusted R² — unitless, R² with a per-feature penalty.
- MAPE — percent. Scale-free, but blows up near zero.
The picture every metric summarizes
Predicted vs actual on the held-out California blocks; y = x is a perfect model.
The vertical gap to the line is each prediction's error.
The math
MSE=n1i=1∑n(yi−y^i)2,RMSE=MSE
MAE=n1i=1∑n∣yi−y^i∣
R2=1−SStotSSres=1−∑(yi−yˉ)2∑(yi−y^i)2
MAPE=n100i=1∑nyiyi−y^i
The outlier demo
Drag one point out; predictions fixed. MSE and RMSE explode, MAE barely moves.
Residual 0 → 12 units: MSE 0.17 → 7.72 units² (45×), RMSE 0.42 → 2.78 units (~7×),
MAE 0.35 → 0.95 units (<3×).
The model's real scores
- RMSE 0.727 $100k — about $72,700 typical error.
- MAE 0.529 $100k — about $52,900. RMSE > MAE means uneven errors.
- MSE 0.528 ($100k)² — uninterpretable; minimize it, don't report it.
- R² 0.585 unitless — explains ~59% of variance. adjusted R² 0.584.
- MAPE 32.06% — off by about a third on a typical block.
Every scratch metric matches sklearn.metrics to the last decimal.
Takeaways
- Report RMSE in the target's units — sensitive and interpretable.
- Keep MAE beside it and read the gap: RMSE ≫ MAE means outliers.
- R² for the scale-free "better than the mean" question, never alone.
- MAPE when a percent is what the room needs and no target is near zero.
- MSE: the best training objective, the worst report-card number.
- Always know what unit you're holding.