QLoRA Explained: Fine-Tune a 7B Model on One GPU with 4-Bit Quantization

A seven billion parameter model sounds small these days. But fine-tune it the standard way and you need over one hundred gigabytes of GPU memory, before you even count the data. That is more than any consumer card, and more than a single eighty gigabyte datacenter GPU. So how do people tune these models on a laptop-class graphics card?

• 7B model, full fine-tune
• Roughly 112 GB before activations

The answer is QLoRA. The intuition is simple. Freeze the big base model and store it in four bit precision, so it takes a fraction of the space. Then train only small low rank adapters, kept in higher precision. The frozen weights are only read, and the tiny adapters carry all the learning.

• Freeze the base, store it in 4-bit
• Train small adapters in higher precision

In the last episode we derived LoRA: instead of updating a big weight matrix W, we learn a low rank update B times A. That saved us gradients and optimizer states for the base model. Today we go one step further and shrink the base model itself, and we will count every byte along the way.

• Last time: LoRA math
• Today: memory and quantized training

To shrink memory, we first need to know where it goes. Training memory has four buckets. The weights themselves. One gradient for every trainable weight. The optimizer states. And the activations saved during the forward pass so backpropagation can use them. Let us take them one at a time.

• Weights
• Gradients
• Optimizer states
• Activations

Memory is just parameters times bytes per parameter. A thirty two bit float takes four bytes. Sixteen bit formats like bf16 take two bytes. A four bit number takes half a byte. So seven billion parameters is twenty eight gigabytes in fp32, fourteen in bf16, and only three and a half in four bit.

• fp32 = 4 bytes
• bf16 = 2 bytes
• 4-bit = 0.5 bytes

Now the optimizer. Adam keeps two running statistics for every trainable weight: a moving average of the gradient, called m, and a moving average of the squared gradient, called v. That is two extra numbers per parameter, and they are usually stored in fp32 for stability. So Adam alone costs eight bytes per trainable parameter.

• First moment m
• Second moment v
• Two extra numbers per weight

Most full fine-tunes use mixed precision. The forward and backward passes run in bf16, which is fast and half the size. bf16 keeps the same exponent range as fp32, so it rarely overflows, but it has fewer mantissa bits, so it is coarse. That is why the optimizer keeps a fp32 master copy of the weights to accumulate small updates accurately.

• Compute in bf16
• Keep fp32 master weights
• bf16 has fp32's range

Assumptions first. Seven billion parameters, bf16 mixed precision, Adam. Weights in bf16: two bytes. Gradients in bf16: two bytes. An fp32 master copy: four bytes. Adam's two fp32 states: eight bytes. That is sixteen bytes per parameter. Times seven billion gives about one hundred twelve gigabytes, and activations come on top.

• 2 + 2 + 4 + 8 = 16 bytes
• 7B x 16 = 112 GB

Now LoRA. The base stays frozen in bf16, so fourteen gigabytes, with no gradients and no optimizer states for it. Say rank sixteen adapters on every linear layer give roughly forty million trainable parameters. That is an approximation, it depends on the architecture. In fp32 with gradients and Adam states, that is sixteen bytes each, about point six four gigabytes. Total, around fifteen gigabytes.

• Base frozen in bf16: 14 GB
• Adapters about 40M params
• Total about 15 GB

Now QLoRA. The frozen base drops to four bits, half a byte per weight, so three and a half gigabytes, plus a little for quantization constants, call it three point six. Add the same point six four gigabytes for adapters. That is roughly four point three gigabytes of static memory, versus one hundred twelve for the full fine-tune.

• Base in 4-bit: about 3.6 GB
• Adapters: about 0.64 GB
• Total about 4.3 GB

Those totals exclude activations, and activations are the wildcard. They grow with batch size, sequence length, hidden size and the number of layers. LoRA and QLoRA do not remove them, because backpropagation still has to flow through the whole network. In practice activations are often what decides whether your run fits.

• Scale with batch and sequence length
• Not removed by LoRA or QLoRA

How do we squeeze a weight into four bits? Sixteen levels only. So we split the weights into small blocks, typically sixty four values. For each block we find the largest absolute value, the absmax, and divide the whole block by it. Everything now lies between minus one and one, and we round each value to the nearest of sixteen allowed levels.

• Split weights into blocks of 64
• Scale each block by its absmax
• Round to 16 levels

Why blocks and not one scale for the whole matrix? Outliers. If a single weight is huge, a global scale would squash every other weight into one or two levels. With small blocks, an outlier only hurts its own sixty three neighbours. The price is that we must store one scale constant per block.

• One outlier ruins a global scale
• Local scales limit the damage

The simplest choice is evenly spaced levels. But trained weights are roughly bell shaped, normally distributed around zero. Most values cluster near the middle, and few sit near the edges. Evenly spaced levels waste precision where there are few weights, and are too coarse where there are many.

• Weights look like a bell curve
• Most values sit near zero

NF4, the four bit NormalFloat, fixes this. It places the sixteen levels at quantiles of a normal distribution, so each level represents about the same share of weights. The levels are dense near zero and sparse near the edges, and they include an exact zero. Because each block is normalized by its absmax, the weights really do fit that shared scale. This is information-theoretically well matched to normally distributed weights.

• Levels at normal-distribution quantiles
• Each level covers equal probability
• Dense near zero, sparse at edges

A tiny example with a block of four weights: zero point four, minus two, one, and zero point two. The absmax is two. Normalized, they become zero point two, minus one, zero point five, and zero point one. Snap to the nearest NF4 levels, roughly zero point one six, minus one, zero point four four, and zero point oh eight. Multiply by two to dequantize: about zero point three two, minus two, zero point eight eight, and zero point one six.

• Block: 0.4, -2.0, 1.0, 0.2
• absmax = 2.0
• Normalize, round, dequantize

Compare with the originals. The largest value came back exactly, because it defined the scale. The others are off by roughly zero point oh four to zero point one two. That is quantization noise. The base model is frozen with this noise baked in, and the adapters are trained on top of it, so they can partly learn to compensate.

• Largest value is exact
• Small rounding error elsewhere
• Noise the adapters can absorb

Here is the key mechanic. The weights are stored in four bit, but matrix multiplication does not happen in four bit. For each layer, the stored codes are dequantized to bf16, used in the matmul, and thrown away. Gradients flow back through that dequantized weight to the adapters. The base weights themselves never get updated.

• Store 4-bit, compute in bf16
• Dequantize on the fly per layer

Trick two is double quantization. Each block of sixty four weights has a thirty two bit scale, which costs half a bit per weight. So we quantize the scales too, to eight bits, in groups of two hundred fifty six. That brings the overhead to about point one two seven bits per weight, saving roughly point three seven bits. On seven billion parameters, that is about a third of a gigabyte.

• Scales cost 32 bits per 64 weights
• Quantize the scales to 8-bit
• 0.5 bits down to about 0.127 bits

Trick three is paged optimizers. Memory use spikes, for example when one very long batch arrives, and a spike can crash an otherwise healthy run. Paged optimizers use unified memory: when the GPU runs out, optimizer state pages are moved to CPU RAM automatically, and brought back when needed. It works like virtual memory in an operating system.

• Memory spikes cause OOM
• Page optimizer states to CPU RAM
• Like virtual memory

Back to activations. Gradient checkpointing stores them only at a few points, say each layer boundary, and throws away the inside of each layer. During the backward pass, it recomputes what it needs by rerunning that layer's forward. You pay roughly one extra forward pass, often around a third more compute, and in return activation memory drops dramatically.

• Save activations only at checkpoints
• Recompute the rest in backward
• Trade compute for memory

Sequence length is the usual killer. Activation memory grows with the number of tokens in the batch, and naive attention stores a score matrix that grows with the square of the length. Doubling sequence length can far more than double memory. Use efficient attention kernels like FlashAttention, cap the length, and lower the batch size while using gradient accumulation.

• Activations grow with tokens
• Attention can grow quadratically
• Use FlashAttention, shorter sequences

Small micro-batches keep memory low, but tiny batches make noisy gradients. Gradient accumulation fixes that. Run, say, a micro-batch of four for eight steps, adding up gradients, and only then update. The effective batch size is four times eight, which is thirty two, while memory stays at the level of a batch of four.

• Micro-batch 1 to 4
• Accumulate gradients over steps
• Effective batch = micro x accumulation

Now the code. With Hugging Face, you create a BitsAndBytesConfig. Set load in four bit to true, quant type to nf4, use double quant true, and, importantly, set the compute dtype to torch bfloat16. Pass that as the quantization config to from pretrained, with device map set to auto. The model loads already quantized.

• BitsAndBytesConfig
• nf4, double quant, bf16 compute

Next, call prepare model for kbit training from PEFT. It casts a few sensitive layers such as norms to fp32 and enables gradient checkpointing. Then define a LoraConfig: rank sixteen, lora alpha thirty two, a small dropout, and target modules covering all the attention and MLP projections. Wrap the model with get peft model, and print the trainable parameters.

• prepare_model_for_kbit_training
• LoraConfig: r, alpha, targets

Training uses TRL's SFTTrainer with an SFTConfig. Set bf16 to true, gradient checkpointing true, a small per device batch size with gradient accumulation, a learning rate around two times ten to the minus four, which is typical for LoRA, and the optimizer to paged adamw eight bit. Pass in your model, dataset and config, then call train.

• SFTConfig and SFTTrainer
• paged_adamw_8bit, bf16=True
• Gradient checkpointing on

Always measure instead of guessing. After loading, print torch dot cuda dot memory allocated divided by one billion to see the static footprint. After a few training steps, print max memory allocated to see the real peak, which includes activations. Compare it with your estimate. If the peak is far above it, suspect sequence length or batch size.

• torch.cuda.memory_allocated
• max_memory_allocated for the peak

After training, you save only the adapter, which is tens of megabytes. To serve, you can load the base and the adapter together. If you merge, merge into a higher precision copy of the base model, not into the four bit weights. Merging into rounded four bit weights adds a second round of rounding error, so load the base in bf16 first.

• Save only the adapter
• Merge into a bf16 base for serving

QLoRA is not free. Every layer must be dequantized on the fly, so each step is slower than plain LoRA on the same hardware. Gradient checkpointing adds more recompute. And because the base is quantized, there can be a small quality gap compared with full precision tuning. The QLoRA paper reported that it largely matched sixteen bit fine-tuning, but always check on your own task.

• Slower steps from dequantization
• Checkpointing adds recompute
• Small quality gap possible

Mistake number one: the wrong compute dtype. If you forget to set it, the dequantized matmuls may run in a slower or heavier format than you intended. Choosing float16 can overflow or give not-a-number losses, because of its small range. Use bf16 on GPUs that support it, and make sure the trainer's bf16 flag matches the compute dtype.

• Default compute dtype can be float32
• float16 may overflow
• Use bfloat16 if supported

A few more traps. Targeting only the query and value projections can leave quality on the table, so target all linear layers. Padding every example to a huge max length wastes activation memory, so cap it sensibly and batch similar lengths together. And remember the merge warning. Finally, if you hit out of memory, reduce sequence length first, then the micro-batch size.

• Too few target modules
• Ignoring padding and max length
• Merging into 4-bit weights

So when should you use what? If you have a cluster and need maximum quality, full fine-tuning is fine. If you have one good GPU and the model fits in bf16, plain LoRA is faster and simpler. If memory is the bottleneck, QLoRA lets you tune a model that would otherwise not fit, accepting slower steps in return.

• Full fine-tune: lots of GPUs
• LoRA: one big GPU
• QLoRA: one small GPU

Let us recap. Training memory is weights, gradients, optimizer states and activations. For a seven billion model, a full mixed precision fine-tune needs about one hundred twelve gigabytes, LoRA about fifteen, and QLoRA about four point three before activations. QLoRA combines NF4 block-wise quantization, double quantization and paged optimizers, with bf16 compute and gradient checkpointing.

• Memory = weights, grads, Adam, activations
• QLoRA: NF4 base, bf16 adapters
• Double quant, paging, checkpointing

So far we have taught models by showing them good answers. But how do you teach a model what people actually prefer? In the next episode we cover RLHF: training a reward model from human comparisons, then optimizing the language model with PPO.

• Supervised tuning teaches format
• RLHF teaches preferences
• Reward models and PPO