Fine-Tune vs RAG vs Prompt: How to Decide (A Practical Framework)

You have an LLM that almost does what you need. It is close, but not close enough. Now you have three tools: rewrite the prompt, bolt on retrieval, or fine-tune the model. Pick wrong and you burn weeks and money. Pick right and you ship in a day.

• Prompt
• RAG
• Fine-tune

Last lesson we derived DPO, which tunes a model on preference pairs without reinforcement learning. That finished our tour of fine-tuning methods. Today there are no new algorithms. Today is about judgment: a framework for deciding which tool to reach for, and in what order.

• Last time: DPO
• This time: choosing

Here is the single question that settles most cases. Is the model failing because it does not know something, or because it does not act the way you want? Missing knowledge and wrong behavior look similar from outside, but they need different cures. Keep that distinction in mind, because everything else builds on it.

• Knowledge or behavior?

Prompting changes only the input. You give instructions, a role, a format, maybe a few examples, as in the few-shot lesson. The weights never move. It is instant to try, free to undo, and you can version it like any text file.

• Instructions
• Examples
• Zero training

RAG also leaves the weights alone. It changes what the model sees by fetching relevant documents at query time and placing them in the prompt. So RAG is a way to supply knowledge: private, fresh, and citable. It does not teach the model a new skill.

• Retrieve documents
• Insert into prompt
• Model stays frozen

Fine-tuning changes the weights themselves, using gradient descent on your examples, whether full or with LoRA. What it teaches well is behavior: tone, format, style, and consistency. Once trained, the behavior is baked in, with no long prompt needed to carry it.

• Gradient updates
• Behavior baked in
• Needs data and evals

Now the axes. First, knowledge versus behavior: knowledge points to RAG, behavior points to fine-tuning. Second, freshness. If facts change daily, weights are the wrong place to store them, because retraining is slow while updating an index is fast. Third, data volume: fine-tuning needs hundreds to thousands of good examples, while a prompt can start with five.

• Knowledge vs behavior
• Freshness
• Data volume

Fourth, latency and cost per call. Long prompts and retrieved context cost tokens on every request. Fifth, privacy. Retrieval can enforce per-user access at query time, while data baked into weights cannot be selectively removed. Sixth, maintenance: prompts need edits, indexes need refreshing, and fine-tuned models need retraining whenever the base model or requirements change.

• Latency and cost
• Privacy
• Maintenance

Seventh, evaluability. Can you measure success? If you cannot, you cannot tell whether any of these tools helped. Eighth, failure modes. Prompts fail by ignoring instructions. RAG fails when retrieval misses or the model ignores the context. Fine-tuned models fail by forgetting, overfitting, or confidently stating stale facts.

• Evaluability
• Failure modes

Put it together as a rough map. Needs fresh or private knowledge: RAG. Needs consistent style, format, or a smaller cheaper model: fine-tune. Unsure or early: prompt. This is a guide, not a law, but it gets you the right first move most of the time.

• Knowledge: RAG
• Behavior: fine-tune
• Start: prompt

Think of a ladder and climb only as far as you must. Step one: write a clear prompt with a system message and an output format. Step two: add examples, which often fixes style and format surprisingly well. Many problems end right here, and you should prove they do not before climbing higher.

• Step 1: Prompt
• Step 2: Examples

Step three: add RAG when the model lacks facts. Your policies, your contracts, yesterday's prices. The facts live in a document store you control, so you can update, delete, and cite them. This is almost always the right home for knowledge.

• Step 3: RAG
• Facts live outside weights

Step four: fine-tune. Reach for it when behavior is the problem, such as a strict house style or a format that must hold on every call. Or when you want distillation: teach a small cheap model to imitate a large model's outputs on your task. This is where fine-tuning really earns its keep.

• Step 4: Fine-tune
• Consistency
• Distillation

Step five: combine. A common pattern is RAG for the facts plus a fine-tuned generator that writes in the right voice and cites sources properly. You can even fine-tune the embedding model for retrieval on your domain's vocabulary. Each layer solves a different problem, so they stack rather than compete.

• Step 5: Combine
• Fine-tuned generator
• Fine-tuned retriever

Let's do cost math. All prices here are hypothetical, chosen for easy arithmetic, so check real prices before deciding. Suppose a large model costs 3 dollars per million input tokens and 15 per million output tokens. Our task runs one million requests, each with a 2000-token prompt packed with instructions and examples, and a 100-token output.

• 1M requests
• Hypothetical prices

Option A, the long prompt on the large model. Input tokens: one million times 2000 is 2 billion, which is 2000 million, times 3 dollars gives 6000 dollars. Output: 100 million tokens, times 15 dollars per million, is 1500 dollars. Total: 7500 dollars, about 0.75 cents per request.

• 2B input tokens
• 6,000 dollars input
• 1,500 dollars output

Option B, a fine-tuned small model. Because behavior is baked in, the prompt shrinks to 200 tokens. Assume the small model costs 0.30 dollars per million input and 1.20 per million output, and fine-tuned serving costs the same. Input: 200 million tokens times 0.30 is 60 dollars. Output: 100 million times 1.20 is 120 dollars. Serving total: 180 dollars.

• Prompt shrinks to 200 tokens
• Small model prices
• Plus training cost

But 180 is not the whole story. Add data labeling, say 5000 dollars of effort, a few training runs at maybe 500 dollars, and engineering time for evals and retraining. Suppose that totals 8000 dollars up front. Then Option B costs about 8180 dollars, which is more than Option A at 7500 for one million requests.

• Data creation
• Training runs
• Evals and retraining

So volume decides. Option A costs 0.0075 dollars per request. Option B costs 0.00018 per request plus the 8000 fixed. Savings per request are about 0.00732. Divide 8000 by that and you get roughly 1.09 million requests to break even. At ten million requests, fine-tuning saves around 65 thousand dollars. At one hundred thousand, it loses badly.

• Savings per request
• Break-even volume

This math ignores some things. A shorter prompt also cuts latency, which users feel. Prompt caching from providers can make long prompts much cheaper, which shifts the break-even point out. And quality matters: a cheaper model that scores worse is no bargain. The lesson is to run your own numbers with real prices and measured quality.

• Quality differences
• Latency gains
• Prompt caching

Case one: legal question answering over a firm's contracts and statutes. This is a knowledge problem. The documents change, access depends on the client, and answers must cite the source clause. RAG is the clear first choice. Fine-tuning would blur facts into weights with no citations and no way to delete a client's data.

• Knowledge-heavy
• Must cite sources
• Changes over time

Case two: a support assistant that must sound like your brand, warm and brief, never defensive. Prompting gets you partway, but tone drifts across thousands of conversations. If you have thousands of approved past replies, fine-tuning on them locks in the voice. Product facts still come from RAG, so this is the combined pattern.

• Brand voice
• Thousands of past replies
• Fine-tune style

Case three: extracting fields from invoices into JSON. Start with schema-constrained output and a few examples from the structured output lesson. If that hits your accuracy target, stop. If volume is huge and you need the cost down, distill into a small model fine-tuned on the large model's verified outputs. The task is narrow, which small models handle well.

• Strict format
• High volume
• Small model suffices

Case four: generating code that follows your team's conventions. Begin by putting the style guide and a few exemplary snippets in the prompt, and retrieve relevant files from your repository. If the model still violates conventions in a stubborn, consistent way, and you have a large corpus of reviewed code, a light LoRA fine-tune can internalize the style.

• Internal conventions
• Style guide in prompt
• Fine-tune if consistent

Notice the pattern. Facts that change or need citing get retrieved. Form that must be consistent gets tuned. And in every case, the cheap options came first, so the expensive one had to prove it was needed.

• Facts: retrieve
• Form: tune
• Try cheap first

Let's encode this as a tiny script. Define a function that takes booleans such as needs fresh knowledge, needs citations, needs strict format, has more than one thousand labeled examples, and high volume. Then it returns a list of recommendations. If fresh knowledge or citations are needed, add RAG. If strict format or style is needed and you have enough examples, add fine-tuning. If volume is high and a small model could do it, suggest distillation. Always begin with prompting and examples.

• Plain Python checklist
• Answers in, advice out

The helper only suggests. Evidence decides. Reuse the evaluation scaffold from earlier lessons: a fixed test set of inputs with expected outputs, a scoring function, and a list of variants. Each variant is just a function from input to output: baseline prompt, prompt with examples, RAG pipeline, fine-tuned model. Loop over them, score each, and record accuracy, average latency, and cost per request.

• Same test set
• Same metric
• Compare variants

Read the table as marginal gains per rung. Suppose the plain prompt scores 72 percent, with examples 81, with RAG 90, and with fine-tuning 92. That last two points came with extra cost and maintenance, so ask whether they are worth it. These numbers are illustrative, not a benchmark. Your own test set, built from real inputs, is the only number that counts.

• Gain per rung
• Is it worth it?

The first big mistake: fine-tuning to teach facts. Fine-tuning on a few documents rarely gives reliable recall. The model absorbs the style of the text more than the details, and can produce fluent answers with wrong specifics. Even if it works, you cannot cite, update, or delete. Put facts in a retrieval store.

• Weak recall
• Hallucinated details
• No citations, hard to update

The second mistake: skipping evals. Without a test set, you judge by trying a handful of examples and feeling good. A fine-tune that improves tone may quietly break reasoning or format. Build the test set first, before any training, and rerun it after every change.

• Vibes are not metrics
• Regressions go unseen

The third mistake: fine-tuning too early. If your prompt is weak, your requirements are still shifting, or your training data is noisy, you will bake in the confusion. A fine-tune on messy labels learns messy behavior. Also, base models improve fast, so a good prompt on a newer model can beat an older tuned one. Fine-tune once the task is stable.

• Prompt not yet tried well
• Requirements still shifting
• Data quality low

A few smaller traps. Blaming the generator when retrieval is the broken part, which the RAG evaluation lesson covered. Ignoring that a model upgrade can change everything, so keep your evals ready. And training for many epochs on a tiny dataset, which memorizes it instead of generalizing. Hold out validation data and watch the gap.

• Judging RAG without retrieval metrics
• Ignoring model upgrades
• Overfitting small data

Here is the whole flow. One: define the task and build an eval set. Two: baseline with a clear prompt and examples. Three: look at the failures and label each one as missing knowledge or wrong behavior. Four: fix knowledge failures with retrieval, behavior failures with fine-tuning. Five: re-measure, and stop when you meet the target.

• Define eval
• Baseline prompt
• Diagnose the gap

Diagnosis is simple in practice. Read twenty failures. If the answer has a wrong or missing fact, and the right fact was never in the prompt, that is knowledge. If the right fact was in the context but the model ignored it or formatted badly, that is behavior or retrieval ranking. Counting these categories tells you which rung to climb.

• Wrong fact? Knowledge
• Right fact, wrong form? Behavior

Let's recap. Prompting changes the input, RAG changes the context, fine-tuning changes the weights. Use RAG for knowledge, fine-tuning for behavior, consistency, and distillation. Climb the ladder only as far as your evals require, and do the cost math with real prices and real volume.

• Prompt first
• RAG for knowledge
• Fine-tune for behavior
• Measure everything

That closes our fine-tuning section: SFT, LoRA, QLoRA, RLHF, DPO, and now how to choose. Next time we begin agents with tool and function calling, where a model stops just answering and starts acting by calling your code.

• Fine-tuning section complete
• Next: tool calling