Supervised Fine-Tuning Explained: Chat Templates, Loss Masking and Code

Ask a freshly pretrained language model, what is the capital of France, and you may get back another question, then a list of quiz questions, then a random forum post. It is not broken. It is doing exactly what it was trained to do, which is continue text. So how does that become an assistant?

• Prompt: capital of France?
• Base model rambles on
• It only continues text

Today is supervised fine-tuning, usually shortened to SFT, and instruction tuning. We will look at the data, the chat template, and the masked loss that makes the whole thing work, with real numbers. Then we will fine-tune a small model in code, evaluate it, and cover the mistakes that quietly ruin runs.

• Data and chat templates
• Masked loss, derived
• Code, evaluation, pitfalls

Recall from the ML course that pretraining trains a Transformer on huge text with next-token prediction, and fine-tuning simply continues training on a smaller, targeted dataset. In the previous episode we evaluated RAG systems. Now we change the model itself. The remarkable part is that SFT uses the same cross-entropy loss as pretraining. Only the data and the masking change.

• Pretraining: next-token on web text
• Fine-tuning: continue training
• Same loss, new data

A base model has read documents, not conversations aimed at helping you. Given a question, the most probable continuation on the internet is often more questions, or a forum reply that is wrong or rude. The knowledge is inside the model. What is missing is the behavior of reading a request and answering it.

• Training data: documents
• Most likely continuation
• Not the helpful continuation

SFT works by showing the model many examples of the behavior we want. A request goes in, an ideal response comes out. The model is not learning much new knowledge here. It is learning which of the many things it can already do it should do when a user asks, and in what style.

• Knowledge comes from pretraining
• SFT teaches the format
• Show, don't tell

Instruction tuning is SFT on a broad mix of tasks, all written as instructions: summarize this, translate that, write a function, explain the idea. Training on many varied tasks teaches the model the general skill of following instructions, so it can handle instructions it never saw during training.

• Many tasks, one format
• Instruction in, response out
• Generalizes to new instructions

Formally, we have pairs of a prompt x and a desired response y. We want the model to assign high probability to y given x. That means minimizing the negative log likelihood of the response tokens, each conditioned on the prompt and on the response tokens before it. It is ordinary language modeling, aimed at a specific slice of text.

• Maximize log probability of responses
• Condition on the prompt
• Standard next-token loss

An instruction dataset is a list of examples. In the simplest form each has an instruction and a response. Modern chat data is a list of messages, each with a role, system, user, or assistant, and some content. Usually this is stored as one JSON object per line, and quality matters far more than the file format.

• Instruction / response pairs
• Multi-turn conversations
• Often JSON lines

A Transformer only sees a flat sequence of tokens. It has no idea what a role is. So we need a chat template, a fixed recipe that turns a list of messages into one string, using special tokens as markers for who is speaking and where each turn ends.

• Model sees one flat token stream
• Roles must be marked
• Template turns messages into text

Different model families use different markers, but the idea is the same. A token marks the start of a user turn, another starts the assistant turn, and an end-of-turn token closes it. That last marker is crucial. During SFT the model learns to produce the end token, which is how it learns to stop talking instead of rambling forever.

• Start-of-turn markers
• End-of-turn token
• Model learns when to stop

In Hugging Face, the tokenizer carries the model's own template. You call apply chat template on your list of messages, and it returns the exact formatted string or token ids. Always print one example and read it. Seeing the raw markers once will save you from hours of debugging later.

• tokenizer.apply_chat_template
• Use the model's own template
• Inspect the output string

The template used in training must match the one used at inference. If you train with one set of markers and prompt with another, the model sees a format it was never taught. Within your data, keep style consistent too. Mixed formats, random casing of roles, or inconsistent answer styles teach the model noise instead of a clear behavior.

• Train and inference templates match
• Same markers everywhere
• One style per dataset

Now the key idea. In a training sequence, the user's prompt is input that will be given at inference time. The model never has to generate it. Only the assistant tokens are what we want the model to produce. So we compute the loss only on assistant tokens and ignore the rest. This is called loss masking.

• Prompt tokens: given input
• Assistant tokens: target behavior
• Train only on the target

Let the sequence have tokens x one through T. Define a mask m sub t equal to one for assistant tokens and zero otherwise. The loss is minus the sum over t of m sub t times log p of x sub t given the earlier tokens, divided by the sum of the masks. The denominator makes it an average over trained tokens only.

• Mask m_t is 0 or 1
• Sum only where m_t is 1
• Divide by count of active tokens

In PyTorch you do not build the mask by hand. You make a labels tensor, copy the input ids, and set every prompt position to minus one hundred. The cross-entropy function has an ignore index that defaults to exactly minus one hundred, so any position with that label contributes nothing to the loss and nothing to the average.

• labels = copy of input_ids
• Prompt positions set to -100
• ignore_index skips them

Take ten tokens, six of prompt and four of answer. Suppose the per-token losses on the answer are two, one, one half, and one half. They sum to four, and dividing by four active tokens gives a loss of one. Now suppose the six prompt tokens each had a loss of three. Including them adds eighteen, for a total of twenty two over ten tokens, or two point two. That is a very different signal.

• Assistant losses: 2.0, 1.0, 0.5, 0.5
• Sum 4.0, mean 1.0
• Prompt losses ignored

One subtlety. The logits at position t predict the token at position t plus one. Hugging Face models shift the labels internally, so you pass labels aligned with the input ids, not pre-shifted. The first token in a sequence is never predicted, which is fine. Just do not shift twice, or your model learns to predict the wrong token.

• Position t predicts token t+1
• Labels aligned with shift
• Model handles it internally

Why not train on everything? Prompts often contain long documents or boilerplate. Training on them spends gradient on memorizing text the model never needs to write, and it can nudge the model to imitate user phrasing. Masking focuses learning on the behavior we care about. Some research reports mixed results, so treat it as a strong default rather than a law.

• Prompts are not our target
• Avoid memorizing user text
• Focus gradient on behavior

Published work on instruction tuning has shown that a small set of carefully curated, diverse examples can produce surprisingly capable assistants. The lesson is that the model imitates what you show it, mistakes included. A thousand clean examples usually beat fifty thousand sloppy ones. Always read a random sample of your data before training.

• Small clean sets can work
• Noisy data teaches noise
• Review samples by hand

A good example has a clear instruction, a correct and complete answer, and the tone you want in production. Diversity matters too. Vary tasks, lengths, and phrasings. Remove duplicates, empty answers, and truncated responses. A truncated answer without its end token teaches the model to stop mid sentence.

• Correct and complete
• Diverse tasks and lengths
• Matches your target style

Many teams generate training data with a stronger model, giving it seed tasks and asking for instructions and responses. This is cheap and scalable, but errors and quirks get copied, and diversity can collapse into repetitive patterns. Filter aggressively, verify with code or rules where possible, and have humans check a sample.

• Stronger model writes examples
• Cheap and scalable
• Filter and verify

Licensing is easy to overlook. Datasets carry licenses, and some forbid commercial use. Outputs from some hosted models come with terms restricting their use to train competing models. Customer data may contain personal information that you must not train on. This is not legal advice, so read the terms and ask your legal team before shipping.

• Check dataset licenses
• Check model provider terms
• Mind personal data

Before any training, split your data into train, validation, and test sets. Deduplicate across the splits, because near identical examples on both sides make your scores look great while measuring only memory. This is data leakage, and it is one of the most common ways people fool themselves.

• Train / validation / test
• Deduplicate across splits
• Hold out real use cases

Now the code. We use a small model, a few hundred examples, and the Hugging Face stack. Each example is a dictionary with a messages list: a user message and an assistant message. We load it with the datasets library and hold out a validation slice. Small enough for a single modest GPU.

• Tiny instruction dataset
• messages format
• Load with datasets library

The TRL library gives us SFTTrainer. We create an SFT config with a learning rate, often small like one e minus five for full fine-tuning, a few epochs, a batch size, and evaluation each so many steps. Recent versions can compute loss on assistant tokens only, when the template marks them. Check your version's docs for the exact flag.

• SFTConfig: lr, epochs, batch
• assistant-only loss option
• Pass model and tokenizer

If you prefer plain PyTorch, masking is a few lines. Tokenize the prompt part with the generation marker, tokenize the full conversation, copy the ids into labels, and set the first prompt length positions to minus one hundred. Then pass labels to the model and it returns the masked loss. Print the labels once to confirm.

• Tokenize prompt and full text
• labels = ids.clone()
• labels[:prompt_len] = -100

Run training and watch two curves, training loss and validation loss. Early on both fall as the model picks up the format. Later the training loss keeps dropping while validation flattens, then rises. The turning point of validation loss is your signal to stop, or to use the checkpoint just before it.

• Train loss falls steadily
• Validation loss tracks it
• Stop when validation turns up

Training loss alone does not tell you the model is better. Take a fixed set of held-out prompts, generate with the base model and the tuned model using identical decoding settings, and compare. Use a rubric, exact match for structured tasks, or a judge model checked against human review. Always measure the before as well as the after.

• Fixed held-out prompts
• Same decoding settings
• Compare base vs tuned

Fine-tuning moves the weights, and if your data is narrow, skills outside it can degrade. This is catastrophic forgetting. A model tuned only on support tickets may get worse at math or at following unrelated instructions. Reduce the risk with a lower learning rate, fewer epochs, mixing in general data, and by testing a few general tasks alongside your target.

• Narrow data shifts weights
• General skills degrade
• Test general abilities too

Overfitting shows up in a few ways. Validation loss climbs while training loss falls. The model recites training answers word for word. Its outputs become rigid and repetitive, even when the question calls for something different. With small datasets this can happen within a couple of epochs, so keep epochs low and check often.

• Validation loss rising
• Verbatim training answers
• Repetitive, rigid style

The first mistake is the wrong chat template. Hand writing markers instead of using the tokenizer's template, using a template from another model family, or adding the beginning token twice. Symptoms are odd output, endless generation, or strange tokens leaking into replies. Always print a formatted example and compare it with the model card.

• Mismatch with the base model
• Missing end-of-turn token
• Double special tokens

The second mistake is training on prompts by accident, usually because labels were never masked, so print your labels and look for minus one hundred. The third is leakage, where test examples or near duplicates sit in the training data. Your scores will look wonderful and mean nothing. Deduplicate, split first, and test on data that resembles real usage.

• Training on prompts by accident
• Data leakage into the test set
• Print labels and dedupe

Let us recap. Supervised fine-tuning turns a base model into an assistant by training on prompt and response examples with the usual cross-entropy loss. A chat template marks the roles and the end of each turn, and masking with minus one hundred means only assistant tokens are learned. Data quality, honest evaluation, and watching for forgetting and overfitting decide whether it works.

• SFT: same loss, curated data
• Template plus masked loss
• Quality, evaluation, forgetting

Everything today updated all the model's weights, which gets expensive fast for large models. In the next episode we derive LoRA, low rank adaptation, which trains a tiny set of extra parameters and freezes the rest.

• Full fine-tuning is costly
• Train small low-rank adapters
• Episode 22: LoRA math