LoRA Math Explained: Low-Rank Adaptation Derived From Scratch

What if you could fine-tune a giant language model by training less than half a percent of the numbers in each weight matrix, and still get most of the quality? That is the promise of LoRA, and it is not magic. It is a small, elegant piece of linear algebra. In this lesson we derive it from scratch, count the parameters exactly, and then code it.

• Full matrix: 16,777,216 numbers
• LoRA: 65,536 numbers

In the previous lesson we covered supervised fine-tuning and instruction tuning, where every weight in the model gets updated. That works, but each parameter needs a gradient and optimizer state too, so memory balloons to several times the model size. As in the ML course's PEFT overview, we want to update far fewer numbers. Today we derive the most popular way to do it.

• Lesson 21: supervised fine-tuning
• Full fine-tuning is expensive

Here is the plan. First the low rank hypothesis, which is the idea that makes everything possible. Then the formula, the initialization, and the gradients. Then the exact parameter count, the design choices like rank and alpha, runnable PyTorch code, and finally the mistakes and limitations you should know about.

• Hypothesis and formula
• Init, gradients, merging
• Code, pitfalls, limits

Quick reminder of rank. A matrix of rank one is an outer product: a column vector times a row vector, so every row is just a scaled copy of the same pattern. Rank two is a sum of two such outer products, and so on. A matrix can be huge, yet have a small rank, meaning it only really contains a few independent directions of information.

• Rank = independent directions
• Rank 1: one outer product

A d by k matrix of rank r can always be written as a tall thin matrix, d by r, times a short wide matrix, r by k. The small dimension r is a bottleneck. Instead of storing d times k numbers, you store r times d plus k numbers. When r is tiny compared to d and k, the savings are enormous.

• Big matrix, small rank
• Can be stored as two thin factors

Now the key hypothesis. The pretrained weight matrix is usually full rank and rich. But the change we need for a specific task, the update delta W, is believed to have low intrinsic rank. Intuitively, the model already knows language; adapting it to a new style or task only needs to nudge it along a few directions. This is a hypothesis supported by experiments, not a theorem.

• Pretrained weights: full rank
• Updates: low intrinsic rank

Think of a sculptor. Pretraining carves the whole statue. Fine-tuning adds a few finishing touches. Those touches do not need thousands of independent directions. Earlier work on intrinsic dimension showed that fine-tuning large models can succeed inside surprisingly small subspaces, and LoRA turns that observation into a practical method.

• Pretraining did the heavy lifting
• Tasks need small nudges

Here is the formula. The adapted weight, W prime, equals the original weight W plus an update delta W. We constrain delta W to equal alpha divided by r, times B times A. W stays frozen. Only B and A are trained. Everything else about the network is untouched.

• W' = W + delta W
• delta W = (alpha / r) B A

Define the symbols. W is d by k and frozen. B is d by r. A is r by k. The rank r is a small integer, such as 4, 8, or 16. Alpha is a scalar hyperparameter. Multiplying B by A gives a d by k matrix, exactly the shape of W, so the sum is well defined.

• B: d by r, A: r by k
• r much smaller than d and k

In the forward pass, input x takes two paths. The frozen path computes W times x. The adapter path first multiplies by A, squeezing x down to just r numbers, then multiplies by B, expanding back to d numbers, and scales by alpha over r. The two results are added. Notice we never build the full delta W during training.

• h = Wx + (alpha/r) B(Ax)
• Never build the full delta W

A tiny example with rank one. Let x be one, two, zero, minus one. Let A be a single row of four values, each 0.5. Then A times x is 0.5 times the sum one plus two plus zero minus one, which is 1. Let B be the column 0.1, minus 0.2, 0.3, 0, and the scale be 2. The adapter output is 2 times 1 times that column: 0.2, minus 0.4, 0.6, and 0.

• x = [1, 2, 0, -1]
• A x = 1, then B scales it

Now the exact count. Take one square weight matrix of size 4096 by 4096, typical in a 7 billion parameter model. Full fine-tuning updates every entry: 4096 times 4096, which equals 16,777,216 trainable parameters, nearly 17 million, for just this one matrix.

• 4096 x 4096
• = 16,777,216 parameters

With LoRA at rank 8, B is 4096 by 8, which is 32,768 numbers. A is 8 by 4096, also 32,768 numbers. Total: 65,536 trainable parameters. In general the count is r times d plus k, which here is 8 times 8192, the same 65,536.

• B: 4096 x 8 = 32,768
• A: 8 x 4096 = 32,768
• Total: 65,536

Divide 65,536 by 16,777,216 and you get exactly one over 256, about 0.39 percent. So this adapter is 256 times smaller than the full update. Because the count grows with r times d plus k, instead of d times k, doubling the rank only doubles the adapter, while full size stays fixed.

• 65,536 / 16,777,216 = 1/256
• About 0.39 percent

Scale it up. Suppose a model has 32 layers with hidden size 4096, and we adapt the query and value projections, both 4096 by 4096. Per layer that is 2 times 65,536, which is 131,072. Across 32 layers, 4,194,304 trainable parameters, roughly 4 million, against billions frozen. That also shrinks gradients and optimizer states, which is where the memory savings come from.

• 32 layers, q and v projections
• 2 x 65,536 x 32 = 4,194,304

Initialization matters because we want training to begin exactly at the pretrained model. So A is initialized randomly, typically with a Kaiming or Gaussian scheme, and B is initialized to zero. Then B times A is the zero matrix, delta W is zero, and the adapted model initially behaves identically to the original.

• Start exactly at the pretrained model
• A random, B zero

Why not set both to zero? We will see in a moment that both gradients would be zero, so nothing would ever move. Why not both random? Then delta W starts as random noise, which perturbs the pretrained model before it has learned anything. Random A with zero B gives a clean start and a nonzero learning signal.

• Both zero: no learning signal
• Both random: noisy start

Let s be alpha over r, let z equal A times x, and let h equal W x plus s times B times z. Suppose the loss sends back a gradient g, the derivative of the loss with respect to h. Now apply the chain rule through the adapter path, from the output backwards into B, and then into A.

• z = A x, h = W x + s B z
• Let g = dL/dh

The gradient for B is s times g times z transposed, an outer product of the output gradient and the squeezed input. The gradient for A is s times B transposed times g, then an outer product with x transposed. Notice that the A gradient contains B. At the start B is zero, so A gets no gradient on step one, but B does.

• dL/dB = s g z^T
• dL/dA = s B^T g x^T

Use our earlier numbers. Say g is 1, minus 1, 0.5, 0, with z equal to 1 and s equal to 2. The gradient for B is 2 times 1 times g, giving 2, minus 2, 1, 0. The gradient for A is zero on this first step because B is zero. After one update, B becomes nonzero, and from the second step on, A learns too.

• g = [1, -1, 0.5, 0], z = 1, s = 2
• dL/dB = [2, -2, 1, 0]

Why the scale alpha over r? As rank grows, the sum B A has more terms, and its magnitude would grow. Dividing by r keeps the update size roughly steady, so you can change the rank without retuning the learning rate from scratch. Alpha is then a single knob that controls how loudly the adapter speaks relative to the base weights.

• Keeps update magnitude stable
• Retune less when r changes

A big practical benefit: merging. After training, compute W merged equals W plus alpha over r times B A, once, and save that. The result has exactly the shape of the original matrix, so inference runs the original architecture with no extra layers and no added latency. Compare that with some adapter methods that insert extra modules and slow the forward pass.

• W_merged = W + (alpha/r) B A
• Zero extra inference latency

You can also keep the adapter separate. One frozen base model can serve many tasks by swapping small adapter files, each often just megabytes instead of many gigabytes. A customer support adapter, a legal adapter, a code adapter, all share the same base weights in memory. Merge when you want speed and simplicity; keep separate when you want flexibility.

• One base model, many adapters
• Adapter file: megabytes

Which modules should get adapters? The original paper focused on the attention query and value projections. In practice, many practitioners now adapt all the linear layers, including key, output, and the MLP projections, because it usually improves quality for the same rank. The cost is more trainable parameters, but still tiny compared with the model.

• Attention: q, k, v, o
• MLP: gate, up, down

How do you pick rank? Common values run from 4 up to 64. Quality often rises quickly and then flattens, while cost grows linearly. A sensible workflow is to start with 8 or 16, check validation loss, and raise rank only if the model underfits. Simple style or format tasks need little rank; learning genuinely new knowledge may need more.

• Typical r: 4 to 64
• Start small, raise if underfitting

For alpha, popular defaults are alpha equal to r, or alpha equal to twice r. With r equal 8, alpha 16 gives a scale of 2, and alpha 32 gives a scale of 4. A bigger scale makes the adapter's update count for more, which behaves a lot like a larger learning rate for the adapter. Tune it together with the learning rate, not in isolation.

• Common: alpha = r or 2r
• Effective scale = alpha / r

LoRA dropout applies standard dropout to the input of the adapter path only, not to the frozen path. It acts as a regularizer, helpful on small datasets where the adapter could overfit. Values from zero to about 0.1 are typical. With lots of data, you can often leave it at zero.

• Dropout on adapter input
• Typical values: 0.0 to 0.1

Now code. Write a LoRALinear module that wraps an existing nn.Linear. In the constructor, freeze the base layer by setting requires_grad to false. Create A as a parameter of shape r by in features, with small random values, and B as zeros of shape out features by r. Set scale to alpha over r. The forward pass returns base of x, plus scale times dropout of x, times A transposed, times B transposed.

• Freeze base, train A and B
• Zero-init B

Test it. Wrap a 4096 by 4096 linear layer with rank 8. Before training, the output should match the base layer exactly, because B is zero. Counting only parameters that require gradients gives 65,536. The total is higher, 16,842,752, because the frozen layer also has 4096 bias values plus the adapter parameters. Always check this count; it catches freezing mistakes immediately.

• Output equals base at init
• Trainable params: 65,536

In real projects use the Hugging Face PEFT library. Create a LoraConfig with r equal 8, lora alpha 16, dropout 0.05, target modules q proj and v proj, and task type causal language model. Call get peft model on your loaded model, then print trainable parameters. It reports the trainable count and the percentage of the total, so you can check it against your own hand calculation.

• LoraConfig: r, alpha, dropout, targets
• get_peft_model(model, config)

After training, call save pretrained to store only the adapter. To reuse it, load the base model, then load the adapter on top with PeftModel from pretrained. For deployment, call merge and unload, which folds B A into W and returns a plain model with no adapter overhead. Save that merged model if you want a standalone checkpoint.

• save_pretrained: adapter only
• merge_and_unload() for deployment

The first common mistake is wrong target modules. Layer names differ between architectures. If your names do not match, you either get an error or adapt far less than you intended. Print the model's module names, set targets deliberately, and always confirm that the trainable parameter count is what you expect.

• Names differ per architecture
• Check trainable count is not zero

The second mistake is a learning rate mismatch. Full fine-tuning uses small rates, like one e minus five. Copy that to LoRA and the tiny adapter barely moves. LoRA commonly works with rates around one e minus four to three e minus four. Too high, though, and training oscillates, especially with a large alpha. Treat alpha and learning rate as a pair.

• LoRA likes higher LR than full FT
• Often around 1e-4 to 3e-4

The third mistake is forgetting the adapter at inference. If you save only the adapter and then load just the base model, you will see base behavior and wonder why fine-tuning did nothing. Always load the adapter or merge it. Also remember, merge into the same precision and base checkpoint you trained against.

• Base model alone gives base behavior
• Load or merge before serving

Now the honest limitations. LoRA constrains the update to low rank, so on some tasks, especially those needing large shifts in behavior or lots of new knowledge, it can underperform full fine-tuning. Higher rank and wider module coverage help, but not always. On many common instruction and style tasks, though, it is competitive, and that is why it is so widely used.

• May trail full fine-tuning on some tasks
• Hard for large new knowledge

Recap. Fine-tuning updates are assumed to have low intrinsic rank, so we write delta W as alpha over r times B A, with B of shape d by r and A of shape r by k. A starts random and B starts at zero, so training begins at the pretrained model. For a 4096 square matrix at rank 8, that is 65,536 parameters instead of 16,777,216. And after training we can merge the adapter into W with no inference cost.

• delta W = (alpha/r) B A
• B zero, A random
• 65,536 vs 16,777,216

LoRA shrinks what we train, but the frozen base weights still take memory. In the next lesson, QLoRA, we quantize the frozen base model down to four bits and train LoRA adapters on top, fitting big models on a single GPU.

• Quantize the frozen base to 4 bits
• Train LoRA on top