Pretraining, Fine-Tuning and LoRA Explained With Real Numbers

A modern language model costs a fortune to train, yet you can adapt one to your own task on a single graphics card in an afternoon. How is that possible? The answer is a two stage idea: pretrain once on a huge pile of text, then adapt cheaply many times. This lesson shows exactly how both stages work, with real numbers.

• Pretrain once
• Adapt many times

In lesson 56 we assembled the full transformer block, with attention, residual connections and layer norm. But an architecture is just an empty machine with random weights. Today we answer how those weights get their values, how they get specialised, and how to change them without paying the full price.

• Lesson 56: full block
• Today: how it learns

We will go in six steps. First the training objectives, including perplexity worked by hand. Then scaling, full fine-tuning and instruction tuning, followed by preference alignment. After that comes the star of the lesson, LoRA, derived with parameter counts, and we finish with a coding demo and the traps that catch beginners.

• Objectives and scaling
• Fine-tuning and alignment
• LoRA, QLoRA, traps

The most common pretraining task is next token prediction. Given the tokens so far, the model outputs a probability for every token in the vocabulary, and the true next token is the answer. Notice that the labels come free from the text itself, so no human annotation is needed. This is called self supervised learning, and it is why we can train on enormous corpora.

• Predict the next token
• Self-supervised labels

Recall from lesson 10 that cross entropy with a one hot target reduces to the negative log of the probability given to the correct class. So the loss at one position is minus log p of the true token. Average this over all T positions in a sequence and you have the training loss. Minimising it is exactly maximum likelihood on the text, which ties this back to what we learned earlier.

• Loss = minus log p(true token)
• Average over positions

Let us compute perplexity. Suppose the model gives the three true tokens probabilities of one half, one quarter and one eighth. The negative natural logs are about zero point six nine three, one point three eight six and two point zero seven nine, and their average is one point three eight six. Perplexity is the exponential of that average, which equals exactly four.

• Probabilities 0.5, 0.25, 0.125
• Mean loss ln 4 = 1.386
• Perplexity = exp(loss) = 4

Perplexity has a friendly reading: it is the effective number of equally likely choices the model is torn between. A perplexity of four means it behaves as if picking among four tokens at each step. A model that guesses uniformly over a vocabulary of V tokens has perplexity exactly V. A common mistake is comparing perplexities across models with different tokenizers, which is unfair because the tokens themselves differ.

• Effective branching factor
• Uniform guess over V gives V
• Lower is better

Remember the causal mask from lesson 56. Because each position can only look backwards, one forward pass over a sequence of T tokens produces T separate predictions at once, each with its own loss term. That is a big reason decoder only models train so efficiently. Every token in the data becomes a training example.

• One pass, T predictions
• Mask hides the future

The second major objective is masked language modeling, used by BERT style encoders. We hide a fraction of tokens, often around fifteen percent, and ask the model to fill them in using context from both the left and the right. The loss is cross entropy computed only at the masked positions. Because it sees both directions it builds rich representations for understanding, but it cannot generate text left to right as naturally as a causal model.

• Hide about 15 percent of tokens
• Predict them using both sides
• Loss only on masked slots

So which one should you pick? Next token prediction suits generation, and it scales into today's chat models. Masked prediction suits tasks like classification and search embeddings. Notice the shared core: both are cross entropy over a vocabulary, and only the pattern of what is hidden changes.

• Causal: generation
• Masked: understanding
• Same cross-entropy core

One of the most striking empirical findings is that pretraining loss tends to fall smoothly and predictably as we scale up parameters, data and compute. On a log log plot, the curves look close to straight lines. That predictability lets labs forecast the result of a huge run by fitting small ones first. I will stay general here, because exact constants depend on the setup and I do not want to quote numbers I cannot verify.

• More parameters, data, compute
• Smooth, predictable curves

Given a fixed compute budget, you must split it between a bigger model and more training tokens. Research on compute optimal training showed that many early large models were undertrained: too many parameters for too little data. The lesson is balance, since both model size and dataset size should grow together. Training compute grows roughly with parameters times tokens, so doubling both is a fourfold bill.

• Big model, little data wastes
• Tokens must grow with parameters

The demands are enormous. Pretraining corpora are measured in huge numbers of tokens, and training runs use large clusters of accelerators for weeks. Yet raw size is not everything. Filtering, removing duplicates and balancing sources strongly affect quality, so data engineering is as important as architecture. This is why pretraining is something most of us download rather than repeat.

• Trillions of tokens possible
• Thousands of accelerators
• Quality and deduplication matter

A caution about scaling laws: they describe smooth loss, not guaranteed skills. Task performance can look flat and then jump, though researchers debate how much of that is real and how much comes from the metric used. Also, extrapolating beyond the range you measured is risky. Treat scaling laws as a planning tool, not a prophecy.

• Loss is not ability
• Emergent behaviour is debated

Transfer learning means starting from weights that already know a lot about language, instead of from random numbers. The pretrained model has learned grammar, facts and patterns, so your small dataset only needs to nudge it. In terms of optimisation, you begin already inside a good basin of the loss landscape. That is why fine tuning converges fast and works with far less data.

• Start from pretrained weights
• Small data, small learning rate

In full fine tuning we continue gradient descent on every parameter using task data, usually with a much smaller learning rate than pretraining, such as one in a hundred thousand. A small rate protects what the model already knows. The cost is memory: with a common mixed precision Adam setup you need roughly sixteen bytes per parameter for weights, gradients and optimizer states. For a seven billion parameter model that is on the order of one hundred twelve gigabytes.

• Update every weight
• Learning rate around 1e-5
• Same loss, new data

A pretrained model completes text, but it does not necessarily follow instructions. Instruction tuning fixes that by fine tuning on many pairs of an instruction and a good response, across varied tasks. The loss is still next token cross entropy, but usually computed only on the response tokens, so the model learns to answer rather than to recite the prompt. A frequent bug is forgetting this masking and training on the prompt too.

• Pairs of instruction and answer
• Loss only on the answer
• Teaches following requests

Writing a perfect answer is hard, but picking the better of two answers is easy for a person. Preference alignment exploits that. We collect comparisons, where each prompt has a chosen response and a rejected one, and then steer the model toward the style people prefer, such as helpful, honest and harmless behaviour.

• Good answers are hard to write
• Easy to compare two answers

Reinforcement learning from human feedback has three outline steps. First, train a reward model to score responses so that chosen answers score higher than rejected ones. Second, use reinforcement learning to adjust the language model, called the policy, to earn high reward. Third, add a penalty on the KL divergence from a frozen reference model, so the policy does not drift into gibberish that merely games the reward. We will study the reinforcement learning machinery itself in lessons 59 and 60.

• Fit a reward model
• Optimise policy with RL
• KL penalty to reference

Direct preference optimization, or DPO, shows that you can skip the separate reward model and the reinforcement learning loop. It trains the policy directly on the preference pairs with a simple classification style loss. The loss compares how much more the policy likes the chosen answer than the reference does, against the same quantity for the rejected answer. It is simpler and more stable to run, though it still depends on the quality of the preference data.

• Direct Preference Optimization
• Loss on chosen versus rejected
• Needs reference model

Let us plug in numbers. Define each log ratio as the policy's log probability of a response minus the reference's. Say it is plus two for the chosen answer and minus one for the rejected one, with beta equal to zero point one. The difference is three, times beta gives zero point three, the sigmoid of that is about zero point five seven four, and minus the log is about zero point five five four. Training pushes that margin up, which lowers the loss.

• Beta = 0.1
• Log-ratio chosen +2, rejected -1
• Margin 0.3, loss about 0.554

Full fine tuning has two painful costs. Training needs memory for gradients and optimizer states for every parameter, and each task leaves you with a complete new copy of the model. If you serve ten tasks, you store ten giant models. Parameter efficient fine tuning, or PEFT, asks a better question: can we change only a tiny number of parameters and still get most of the benefit?

• Full tuning copies everything
• One copy per task is huge

LoRA, short for low rank adaptation, rests on a hypothesis: the change needed to adapt a pretrained weight matrix has low intrinsic rank. Recall from lesson 5 that a low rank matrix is a product of thin matrices. So we freeze the original matrix W, and learn its update as the product of two small matrices, B and A. The adapted weight is W plus B times A.

• Updates have low intrinsic rank
• Freeze W, learn a small change

Now the shapes. Suppose W has d rows and k columns. We choose a small rank r, and make B a d by r matrix and A an r by k matrix. Their product B times A is d by k, matching W. The forward pass becomes W x plus B times A x, and the side branch first squeezes x down to r dimensions, then expands it back.

• W is d by k
• B is d by r, A is r by k
• Rank r much less than d

Here is the arithmetic. With d and k both equal to 4096, the full matrix holds 4096 times 4096, which is 16,777,216 parameters. With rank 8, B has 4096 times 8, or 32,768 parameters, and A has the same, for a total of 65,536. That is one part in 256, about zero point three nine percent of the original matrix.

• d = k = 4096: 16,777,216
• r = 8: 65,536 trainable
• Just 0.39 percent

Now apply LoRA to the four attention projections in each of 32 layers. That is 65,536 times 4 times 32, which equals 8,388,608 trainable parameters, about eight point four million. Compare that with billions in the full model. Since the optimizer only tracks these few million, memory for gradients and Adam states nearly vanishes.

• 4 matrices per layer
• 32 layers
• About 8.4 million total

Initialisation matters. LoRA sets B to zeros and A to small random values, so at step zero the product is exactly zero and the model starts identical to the pretrained one. The update is also multiplied by a scale factor, alpha divided by r, which lets you change the rank without retuning the learning rate from scratch. A common mistake is initialising both to zero, which gives zero gradients for both and nothing ever learns.

• B starts at zero
• A starts random
• Scale by alpha over r

After training, you can compute W plus B times A once and store the merged matrix, so the final model has the same shape and no extra layers. Or keep the base model frozen and store each task as a tiny file of B and A matrices, swapping them in as needed. Ten tasks then cost one base model plus ten small files.

• Merge W + BA after training
• No extra layers at runtime
• Swap small adapters per task

LoRA has relatives. Adapters insert small bottleneck layers, which project down, apply a nonlinearity, and project back up, inside each block. Prefix tuning instead learns a handful of continuous vectors that are prepended to the attention keys and values, acting like virtual tokens. All of them freeze the base model. The difference from LoRA is that adapters add a little latency, while LoRA can be merged away.

• Adapters: small bottleneck layers
• Prefix tuning: learned virtual tokens
• Both freeze the base

QLoRA combines LoRA with quantization. The frozen base weights are stored in four bit precision, and only the small LoRA matrices are trained in higher precision. Counting weights alone, seven billion parameters at sixteen bits take about 14 gigabytes, and at four bits about 3.5 gigabytes. That drop is what lets large models be tuned on a single consumer grade card. Quantization costs a little accuracy, so always validate.

• Store frozen weights in 4 bits
• LoRA matrices stay higher precision
• 7B weights: 14 GB to 3.5 GB

Now the demo, using a small model so it runs anywhere. We load a tokenizer and a small causal language model from Hugging Face. Then we create a LoRA configuration with rank 8, alpha 16, targeting the query and value projections, and wrap the model with the peft library. A helper then prints the trainable parameters, which should show well under one percent.

• Load model and tokenizer
• Wrap with LoRA config
• Train with Trainer

Next we format each example as instruction plus response, tokenize it, and set the label of every prompt token to minus one hundred, which tells the loss function to ignore it. We hand the data to the Trainer with a modest learning rate, often around two in ten thousand for LoRA, since fewer weights can tolerate larger steps. While training, watch both training and validation loss. If validation turns upward, you are overfitting.

• Tokenize instruction pairs
• Mask the prompt labels
• Watch train and validation loss

When training finishes, saving the model stores only the adapter, a file of just a few megabytes. To use it, reload the base model and attach the adapter. Finally, run a few held out prompts through both the original and the tuned model and compare. This simple check catches many silent failures, such as a mask bug or a wrong template.

• Save only the adapter
• Reload onto base model
• Compare before and after

Fine tuning has a hidden cost called catastrophic forgetting. As the weights move to fit your narrow data, they can drift away from the settings that supported general abilities, so the model gets better at your task and worse at everything else. Remedies include a lower learning rate, fewer epochs, mixing in some general data, and using LoRA, which limits how far the weights can move.

• New task overwrites old skills
• Gradient steps move shared weights

Evaluation is where many projects go wrong. Pretraining corpora are scraped from the web, so benchmark questions may have been seen already, which is contamination. Repeatedly tuning choices on your test set quietly overfits to it, as lesson 24 warned. Also remember that lower perplexity does not always mean a better assistant, and that a tiny evaluation set gives noisy conclusions. Always keep a clean held out set and measure the abilities you care about, including general ones to detect forgetting.

• Test data leaked into pretraining
• Tuning on the test set
• Perplexity is not quality

So how do you choose? Start with good prompting, since it costs nothing. If that fails, try LoRA, or QLoRA when memory is tight. Reserve full fine tuning for cases where you have lots of data, plenty of hardware and need deep changes. If behaviour or tone is the issue rather than knowledge, preference methods like DPO are worth considering.

• Prompting first
• LoRA for most tasks
• Full tuning when needed

Let us recap. Pretraining minimises cross entropy on next tokens or masked tokens, and perplexity is the exponential of the average loss. Scaling laws say to grow data and model together. Instruction tuning and preference methods like RLHF and DPO turn a text completer into an assistant. LoRA learns a low rank update, W plus B times A, with a tiny fraction of the parameters, and QLoRA shrinks the frozen base further.

• Pretrain: cross-entropy, perplexity
• Align: instruction tuning, RLHF, DPO
• Adapt: LoRA W + BA, QLoRA

In the next lesson we look at what happens after training: inference, latency, memory and the real limits of large language models. Try the demo yourself, change the rank, and watch the trainable parameter count move. See you in lesson 58.

• Lesson 58: inference and limits
• Subscribe for the full course