Capstone: Fine-Tune a Small Open Model End to End (QLoRA + TRL)

Most fine-tuning projects fail before the first training step. Not because of the GPU, or the learning rate, but because nobody wrote down what success means. Today we build a complete fine-tune from start to finish, and we do it in a way that you can trust the result.

• End-to-end pipeline
• Honest results only

This is episode thirty-eight. Last time we built a coding assistant. Now we take what we learned in the fine-tuning lessons, supervised fine-tuning, LoRA, QLoRA and DPO, and put it all into one project with runnable scripts.

• SFT, LoRA, QLoRA, DPO
• One project

Our project is structured extraction. We take messy support emails and produce a strict JSON object with fields like product, issue category, urgency and order number. The same recipe works for domain style or any other narrow task. One important rule: every number you see in your results must come from your own run, so I will show you how to measure, not quote scores.

• Messy text in
• Strict JSON out

Start with the question: should you fine-tune at all? From the earlier lesson on fine-tune versus RAG versus prompt, fine-tuning shines for stable formats, consistent style, and cheaper small models. It is a poor choice for injecting fast-changing facts. Extraction into a fixed schema is a textbook fit.

• Format and style: yes
• Fresh facts: use RAG

Next, pick metrics before you touch a model. For extraction I use three. First, valid JSON rate: does the output parse and match the schema. Second, field-level exact match: for each field, is the value equal to the gold value. Third, a full-record match, where every field must be right. Valid rate is a floor, field match shows partial skill, and record match is the strict number.

• Valid JSON rate
• Field exact match
• Full record match

Here is a worked example with small numbers. Say the test set has ten records with four fields each, so forty fields. The model gets thirty-two fields right: field match is thirty-two over forty, which is zero point eight. If only three records have all four fields right, record match is three over ten, zero point three. Same model, two very different numbers, which is why you report both.

• 32 of 40 fields = 0.8
• 3 of 10 records = 0.3

Now the step people skip: the baseline. Before training anything, run the small base model with a good prompt, a schema description and a few examples. If you do not have this number, you can never say your fine-tune helped. This is mistake number one: no baseline.

• Prompt first
• No baseline, no claim

Here is the shape of the baseline script. It loads the model with the transformers pipeline, builds a prompt with the schema and three examples, runs every test email, and tries to parse the output as JSON. It then calls a scoring function that returns valid rate, field match and record match. Save the raw outputs to disk so you can read the failures later.

• Schema plus few-shot
• Parse, score, save outputs

Then read the failures, do not only look at the score. Typical small-model errors are extra chatty text around the JSON, invented fields, wrong enum values like urgent versus high, and numbers reformatted. These patterns tell you what the training data must teach, and they also suggest whether a better prompt alone could fix things.

• Chatty wrapper text
• Wrong enum values
• Invented fields

Data is where fine-tunes are won. Quality beats quantity: a few thousand clean, consistent examples often beat a huge noisy pile. Start from whatever real labelled emails you are allowed to use, and check privacy: remove names, addresses and anything personal before it goes anywhere near a training file.

• Clean beats big
• Scrub personal data

If you have too few examples, add synthetic ones. Use a strong model to write new emails from a seed: give it a target JSON and ask for a realistic email that implies exactly those values. Generating the text from the label is safer than labelling generated text, because the label is correct by construction. Then still validate every record against the schema and spot-check a sample by hand.

• Label first, text second
• Validate every record

Synthetic data has a trap. If all your emails come from one generator prompt, they share a voice, and the model may learn the voice instead of the task. Mix sources: real examples, several prompt styles, different lengths, typos, and also hard negatives such as emails with missing fields where the right answer is null. Keep real data in your validation and test sets so you measure the real world.

• Several styles
• Hard negatives with nulls
• Real data for testing

Now deduplicate. Exact duplicates are easy: hash the normalized text. Near duplicates need more care, like the same email with a changed order number. A simple approach is to compute a shingle set or an embedding for each email and drop pairs above a similarity threshold. Choose the threshold by reading examples near it, not by guessing.

• Exact hash first
• Near-duplicates by similarity

Splits come next, and the order matters. Deduplicate first, then split, so that near copies cannot land on both sides. A common split is eighty percent train, ten percent validation, ten percent test. If emails come in threads or from the same customer, split by group, not by row, otherwise the model sees the same customer in training and test.

• Dedup before splitting
• Split by group
• 80, 10, 10 as a start

Mistake number two: the contaminated test set. Run an explicit leakage check. Compare every test email to every training email by exact match, by near-duplicate similarity, and by shared order numbers or identifiers. Any hit means remove it from train, not from test. Also check that synthetic examples were not seeded from test labels. A leaked test set gives a beautiful score that means nothing.

• Exact, near, shared IDs
• Remove hits from train

Finally, format the data as chat messages. Each record has a system message with the schema instruction, a user message with the email, and an assistant message containing only the JSON. Save it as JSON lines. Use the same system prompt at training and at inference, because a mismatch there quietly costs accuracy.

• System, user, assistant
• Assistant says only JSON

Time to train. Recall QLoRA from the earlier lesson: the base model is loaded in four-bit precision and frozen, and we train small low-rank adapter matrices in higher precision. That lets a model of a few billion parameters train on a single consumer or free-tier GPU. We use the TRL library, which wraps this in an SFT trainer.

• Frozen 4-bit base
• Trainable low-rank adapters

The setup has three parts. A BitsAndBytes config sets four-bit NF4 quantization with bfloat16 compute. A LoRA config sets the rank, say sixteen, alpha, dropout, and the target modules, usually the attention and MLP projections. An SFT config sets batch size, gradient accumulation, learning rate and epochs. Pass these with the model and dataset to the trainer and call train.

• Quantization config
• LoRA config
• SFT config

Some practical choices. A learning rate around one to two times ten to the minus four is a common starting point for LoRA. Use gradient accumulation to reach an effective batch size you can afford: batch four with accumulation eight is an effective batch of thirty-two. Train on the assistant tokens only if your setup supports it, so the loss focuses on the JSON and not on repeating the email.

• Batch 4 times 8 equals 32
• Mask the prompt in the loss

Watch two curves: training loss and validation loss. If both fall and flatten, good. If training loss keeps dropping while validation loss turns upward, that is overfitting, mistake number three. The model is memorizing your examples. The fixes are fewer epochs, a lower rank or learning rate, more varied data, or early stopping at the best validation checkpoint.

• Train down, val up
• Stop at best checkpoint

Loss is only a proxy. After each run, generate on the validation set and compute your real metrics: valid rate, field match, record match. Use greedy decoding, temperature zero, for extraction, since you want determinism. Keep a small log of each run with its settings and scores, so changes are comparable and you do not fool yourself with memory.

• Greedy decoding
• Log every run

An optional step is a DPO pass, from our earlier lesson on preference optimization without reinforcement learning. When is it worth it? When SFT leaves a specific bad habit, like occasionally adding prose or choosing a wrong enum. You build pairs: a chosen output that is correct, and a rejected output that is a real failure from your own model.

• Chosen vs rejected pairs
• Fix specific bad habits

Mining rejected examples is simple here. Sample from the SFT model on training prompts, score each output with your metric, and keep the failures as rejected, paired with the gold answer as chosen. Run TRL's DPO trainer with a small learning rate and a small beta, then re-evaluate. Keep it only if the held-out metrics improve. If they do not, skip it, and that is a fine result.

• Failures become rejected
• Keep only if it helps

Now the moment of truth, and only now do we touch the test set. Run the same evaluation script on four systems: the base model with a prompt, the same prompt with retrieved examples as a RAG-style variant, the SFT model, and the SFT plus DPO model if you made one. Same inputs, same scoring, same decoding settings.

• Same test, same scorer
• Prompt, RAG, SFT, DPO

Report the table from your own run and nothing else. Include the metrics, the number of test examples, and the cost: latency per request and model size. A fine-tuned small model might match a bigger prompted one at a fraction of the cost, or it might not. Either is a real finding. With a small test set, remember that differences of a point or two can be noise.

• Metrics plus cost
• Small test sets are noisy

Then do error analysis. Group the remaining mistakes by field and by type. If failures cluster on one field, add targeted data for it. If the model fails on long emails, check the max sequence length you trained with. Each round of analysis, data fix and retrain is how real projects improve, and it is usually the data, not the hyperparameters.

• Cluster the failures
• Fix data first

Once you are happy, merge the adapters. The merge computes the new weight as the original weight plus the scaled product of B and A, then saves a single standalone model. Load the base model in half precision, not four-bit, for the merge, then attach the adapter and call merge and unload. Merging in four-bit gives lossy results.

• Merge into half-precision base
• Single standalone model

Merging removes adapter overhead at inference, but the model is still large. Quantize it for serving, as in our quantization lesson: for example GPTQ or AWQ for GPU servers, or a GGUF four-bit file for local runtimes. After quantizing, run the evaluation script again. Quantization can cost accuracy, and you only know by measuring on your own test set.

• GPTQ, AWQ or GGUF
• Measure the quality cost

Finally, save artifacts. Push the adapter, the merged model and the tokenizer to the Hub or to your own storage. Write a model card that states the task, base model and its licence, training data sources and size, the metrics with the test set description, known limitations, and the exact prompt format. Check the licence of your data and base model before you publish, and keep the repo private if you are unsure.

• Adapter, merged, quantized
• Model card with limits

Let us collect the mistakes. No baseline, so you cannot prove improvement. A contaminated test set, so the score is fiction. Overfitting, so the model shines on training data and fails in the wild. Add to those a prompt-format mismatch between training and inference, and tuning on the test set by running it after every change.

• No baseline
• Contaminated test
• Overfitting
• Format mismatch

A related habit is to keep the test set locked. Use validation for decisions and touch test once per major comparison. If you peek and tune on it repeatedly, it quietly becomes a second validation set and you have no honest estimate left. If it happens, collect a fresh test set from new data.

• Decide on validation
• Open test once

Let us recap the pipeline. Define the task and metrics. Baseline with prompting. Build a clean dataset with synthetic augmentation, dedup, group-aware splits and a leakage check. Train with QLoRA and TRL, watch validation loss. Optionally apply DPO. Evaluate against the baselines on the locked test set, merge, quantize, re-evaluate, and publish with a model card.

• Task, baseline, data
• Train, evaluate, ship

The big idea is that fine-tuning is an engineering loop, not a single run. Your edge is not a magic hyperparameter but a clear metric, clean data and honest evaluation. Do those, and a small model you own can do one job very well, cheaply and privately.

• Metric, data, honesty
• Small model, one job

In the next and final capstone, we deploy a tool-using agent: a loop that calls functions, remembers context and runs behind an API, and we can plug in the model you just trained.

• Tool-using agent
• Subscribe for more