One tree is weak. Three trees that fix each other's mistakes are not.
Hook
Learn From Mistakes
What if a model got smarter by only studying its own mistakes? That is gradient boosting.
- Gradient boosting
- Study your own errors
Setup
Start With the Average
Start with a dumb guess: the average of all the data. It misses almost everywhere.
- Simple first guess
- Big errors remain
Round 1
Round 1: Fit Residuals
Round one. Compute the residuals, the leftover errors. Then fit a small tree to predict them.
- Compute residuals
- Small tree predicts them
Add the Tree
Add that tree, scaled by a small learning rate, to the prediction. The line bends toward the curve.
- Scale by learning rate
- Prediction bends closer
Round 2
Round 2: Smaller Errors
Round two. The mistakes are smaller now. Fit a second tree to the new residuals, and add it in.
- Recompute residuals
- Tree 2 fixes more
Round 3
Round 3: Hugs the Curve
Round three. Another tree, another correction. The prediction now hugs the curve.
- Tree 3 corrects again
- Error keeps shrinking
Insight
Residual = Negative Gradient
Here is the secret. For squared error, the residual is exactly the negative gradient of the loss.
- Squared error loss
- Step downhill each round
Payoff
Next: Lesson 42
So boosting is gradient descent, with trees as the steps. Follow for lesson forty-two, where we derive it fully.
- Boosting is gradient descent
- Full math in lesson 42