LoRA Explained: Fine-Tune a Giant Model by Training a Sliver

You can fine-tune a giant model by training only a tiny sliver of it.

• Giant model
• Tiny trainable part

Picture one weight matrix, one thousand by one thousand. That is one million numbers.

• 1000 by 1000
• One million weights

Full fine-tuning updates every single one. That is slow and memory hungry.

• Slow
• Memory hungry

LoRA freezes that matrix. It learns only the change, called delta W.

• W stays frozen
• Learn only delta W

It writes that change as two thin matrices multiplied together. One is one thousand by eight. The other is eight by one thousand.

• B is 1000 by 8
• A is 8 by 1000

Count them. One thousand times eight is eight thousand. Two of them make sixteen thousand.

• 1000 x 8 = 8,000
• Two matrices = 16,000

Sixteen thousand instead of one million. About sixty times fewer. The eight is called the rank, a bet that the useful change is simple.

• 16,000 vs 1,000,000
• Eight is the rank

The frozen weights plus the small update give the adapted model.

• Frozen plus small update

We build it in lesson fifty-seven. Follow to keep learning.

• Build it from scratch