One weight matrix, 16.8 million numbers. LoRA trains just 65,536 of them.
Introduction
LoRA in One Minute
One matrix can hold sixteen million numbers. LoRA trains almost none of them.
- Fine-tune with almost nothing
One Weight Matrix
Take one weight matrix in a big model. Four thousand ninety-six by four thousand ninety-six. That is about sixteen point eight million parameters.
- 4096 by 4096
- About 16.8M parameters
Full Fine-Tuning Is Heavy
Full fine-tuning updates every single one. That means huge gradients and optimizer memory.
- Every weight updated
- Gradients plus optimizer memory
Freeze W, Learn a Side Path
LoRA freezes that matrix instead. Then it learns only a small change on the side.
- W stays frozen
- Learn a small update
The Low-Rank Idea
The idea in one sentence: a big update can be written as the product of two skinny matrices.
- Delta W equals B times A
- Rank r is tiny
Count the Parameters
Use rank eight. Four thousand ninety-six by eight, and eight by four thousand ninety-six. Together, just sixty-five thousand, five hundred thirty-six parameters.
- 4096 x 8 plus 8 x 4096
- Equals 65,536
256 Times Fewer
That is two hundred fifty-six times fewer for this one matrix.
- 16.8M versus 65K
- About 0.4 percent
Next: The LoRA Math
But why does this work? Next lesson, we derive the LoRA math. Follow so you don't miss it.
- Why low rank works
- Follow for the derivation