Every neural network learns with one rule from calculus, and you can see it in 50 seconds.
Introduction
One Rule, Two Gears
Every neural network learns with one rule from calculus, and it fits in two gears.
- The chain rule
x Drives u Drives y
Picture two gears. Turning x moves u. Turning u moves y.
- x moves u
- u moves y
A Numeric Example
Here, u equals x squared, and y equals three times u. Start at x equals two. Then u is four, and y is twelve.
- u = x squared
- y = 3u
- x = 2 gives y = 12
First Gear: Four
Nudge x. At two, u moves four times as fast.
- du/dx = 4
Second Gear: Three
Nudge u, and y moves three times as fast.
- dy/du = 3
Rates Multiply
Gear rates multiply. Four times three is twelve. That is the chain rule.
- dy/dx = dy/du times du/dx
- 3 times 4 = 12
Run It Backward
Now run it backward on a three-node graph. Start at y with a one, multiply by three, then by four. Twelve arrives at x.
- Start with 1 at y
- Multiply 3, then 4
- 12 reaches x
Backprop Is This, Reversed
That backward pass is backpropagation. Same idea, just reversed. We derive it fully in lesson thirty-six. Follow for more.
- Full derivation: lesson 36