Why does Lasso zero out coefficients while Ridge only shrinks them? One picture explains it: a diamond has corners, a circle does not.
Hook
Lasso vs Ridge
Why does Lasso set weights to exactly zero, while Ridge never does?
- Exactly zero?
- Only one does it
Setup
Two Weights
Start with a model that has two weights, w one and w two.
- w1 and w2
- Each point is a model
Loss
Loss Ellipses
The loss draws ellipses around the best fit. Bigger ellipses mean more error.
- Center is best fit
- Outer rings mean more error
Constraint
A Weight Budget
Regularization adds a budget. Weights must stay inside a region around the origin.
- Stay inside the region
- Size set by t
Lasso
Diamond Corners
For Lasso, that region is a diamond. Its sharp corners sit exactly on the axes.
- Corners lie on axes
- On an axis, one weight is zero
Touch
First Touch
Grow the ellipse until it first touches the diamond. It almost always hits a corner. That weight is exactly zero.
- Ellipse meets corner
- w1 equals zero
Ridge
Circle Has No Corners
Ridge uses a circle. It is smooth, with no corners. The touch point slides along the edge, so both weights stay nonzero.
- Smooth edge
- Shrinks, never zero
Payoff
Corners Make Zeros
Corners create zeros. That is why Lasso selects features. Follow for the next lesson.
- Lasso selects features
- Ridge shrinks them