Why does plain gradient descent zig-zag, and how does Adam glide straight to the minimum?
Hook
Why The Zig-Zag?
Why does gradient descent zig-zag like a drunk hiker? And why does Adam walk straight?
- SGD bounces
- Adam glides
Problem
A Narrow Valley
Picture a long, narrow valley. The walls are steep. The floor slopes gently toward the minimum.
- Steep walls
- Gentle floor
Gradient Points Sideways
The gradient points mostly at the nearest wall. So plain SGD bounces across, and crawls along the floor.
- Big sideways push
- Tiny forward push
Momentum
Fix One: Momentum
Fix one: momentum. Average recent gradients. The sideways pushes cancel out. The forward pushes add up.
- Average the direction
- Sideways cancels
- Forward adds up
Scaling
Fix Two: Scaling
Fix two: scaling. Steep directions get smaller steps. Gentle directions get bigger steps. Every parameter gets its own step size.
- Steep: small steps
- Gentle: big steps
- Per-parameter step size
Adam
Adam = Both
Adam combines both. An averaged direction, and a per-parameter step size.
- Averaged direction
- Per-parameter step size
Race
The Race
Watch the race. SGD zig-zags. Momentum curves smoothly. Adam cuts almost straight to the minimum.
- SGD: zig-zag
- Momentum: smooth
- Adam: direct
Payoff
Next: Lesson 47
Lesson forty-seven builds Adam step by step, with the math and the code. Follow, so you do not miss it.
- Math plus Python
- Follow for more