One number decides whether your model learns or explodes. Watch three balls roll down the same loss landscape with three different learning rates.
Hook
One Number Decides Everything
One tiny number decides whether your model learns or explodes.
- The learning rate
Setup
Step Size in Gradient Descent
It is called the learning rate. It controls how big each step is during gradient descent.
- Update: theta minus alpha times gradient
- Alpha sets the step size
Three Balls, Three Rates
Here are three balls on the same landscape. Each starts at the same spot. Only the learning rate changes.
- Same start
- Same landscape
- Different alpha
Small
Too Small: Painfully Slow
The blue ball uses a tiny learning rate. It creeps forward in minuscule steps. It is safe, but painfully slow.
- Tiny steps
- Safe but slow
Just right
Just Right: Converges
The green ball uses a well-chosen rate. Its steps are confident. It slides down the valley and settles at the minimum.
- Confident steps
- Settles at the minimum
Big
Too Big: Overshooting
The red ball takes huge steps. It leaps over the valley, lands higher on the other side, and bounces back and forth.
- Leaps over the valley
- Loss grows each step
Divergence
Soon the red ball flies off the plot entirely. That is divergence. The loss goes to infinity.
- Loss explodes
- Training fails
Payoff
Tune It, Watch the Loss
So try several learning rates and watch the loss curve. Follow for the next lesson in this series.
- Try several rates
- Watch the loss curve