Evaluating RAG: Metrics, Test Sets and 8 Failure Modes

Your RAG demo works beautifully. You ask five questions, you get five fluent answers, and everyone nods. Then real users arrive, and the bot confidently quotes a policy that was retired last year. What went wrong, and how would you have known beforehand? Today we learn to measure RAG properly.

• Looks great in a demo
• Fails on real users

In the last episode we combined BM25 and dense search with reciprocal rank fusion. That gave us a stronger retriever, but stronger according to what? Without measurements, every change is a guess. This lesson closes our RAG module by giving you the metrics, the test sets, and a diagnosis flow for when things break.

• Hybrid search last time
• Now: measuring it all

The key idea is to split RAG into two halves. Retrieval finds passages. Generation writes an answer from them. If the final answer is wrong, the cause could live in either half, and the fixes are completely different. Judging only the final answer is like debugging a program by looking only at its crash message.

• Two separate components
• Two separate failure sources

Think of a two by two grid. Good retrieval and good generation gives a correct answer. Good retrieval with bad generation means the model ignored or misread the evidence. Bad retrieval means the model either refuses, or worse, makes something up. Each cell needs a different repair, so we need metrics that point at the right one.

• Good retrieval, good answer
• Good retrieval, bad answer
• Bad retrieval, bad answer

Retrieval metrics need ground truth: for each test question, which chunks actually contain the answer. Suppose one question has three relevant chunks, called A, C and F. Our retriever returns a ranked list: B, A, D, C, E, and F sits down at position seven. Let's score this list four different ways.

• Labelled relevant chunks
• Ranked list of results

Recall at k asks: of all the relevant chunks, what fraction appear in the top k results? With k equal to five, we found A and C, but not F. That is two out of three, so recall at five is about zero point six seven. Recall is the metric to watch first, because the generator can never use a chunk it was never given.

• Relevant found in top k
• divided by all relevant

Precision at k asks a different question: of the k results we returned, how many were relevant? Still with k equal to five, two of the five are relevant, so precision at five is two over five, which is zero point four. Low precision means noise in the prompt, which costs tokens and can distract the model.

• Relevant in top k
• divided by k

Mean reciprocal rank cares about where the first relevant result lands. Here the first relevant chunk, A, is at rank two, so the reciprocal rank is one half. Say two other questions score one and one quarter. The mean of zero point five, one, and zero point two five is one point seven five divided by three, about zero point five eight.

• Rank of first relevant hit
• Reciprocal: one over rank
• Average across queries

Normalized discounted cumulative gain rewards putting relevant items high, with a logarithmic discount. Each relevant item at position i contributes one divided by log base two of i plus one. Our top five has relevant chunks at positions two and four: zero point six three plus zero point four three, so DCG is about one point oh six.

• Gain divided by log of position
• Normalise by the ideal ranking

To normalize, compute the best possible score. The ideal top five puts all three relevant chunks first: one plus zero point six three plus zero point five, giving about two point one three. Dividing, one point oh six over two point one three gives an nDCG of roughly zero point five. One means perfect ordering, and our retriever is only halfway there.

• Ideal DCG is about 2.13
• nDCG is about 0.50

Use recall at k to decide how many chunks to pass to the model. Use MRR when one good chunk is enough, like simple factual lookups. Use nDCG when relevance has grades, or when order matters for a reranker. A common mistake is reporting only one number; track recall and a rank-aware metric together.

• Recall: did we fetch it
• MRR: how early
• nDCG: graded ordering

Now the generation half. Faithfulness, also called groundedness, asks whether every claim in the answer is supported by the retrieved context. Break the answer into atomic claims. If an answer has five claims and the context supports four, faithfulness is four over five, or zero point eight. The unsupported claim is a hallucination, even if it happens to be true.

• Are claims supported by context
• Claims supported over total claims

Answer relevance asks whether the response actually addresses the question. An answer can be perfectly faithful to the context and still dodge the question, for example by summarizing a whole policy when the user asked about one deadline. Relevance and faithfulness are independent, so you must score them separately.

• Does it address the question
• Not the same as correctness

Two more metrics bridge both halves. Context precision asks how much of the retrieved context was actually useful, ideally with useful chunks ranked first. Context recall asks whether the retrieved context contains everything needed to produce the reference answer. If the reference answer has four facts and the context covers three, context recall is zero point seven five.

• Precision: is context useful
• Recall: is anything missing

Finally, answer correctness compares the generated answer to a gold reference answer. This is the metric stakeholders care about, but it is the least diagnostic, because a wrong answer could come from any component. Treat it as the headline score, and use the other four metrics as the instruments that explain it.

• Compare to a reference answer
• Needs gold labels

Checking claims by hand does not scale, so teams use a strong LLM as a judge. You give it the question, the context, and the answer, then ask it to list claims and mark each as supported or not. This is fast and cheap enough to run on every change, but a judge is a model, with all the usual flaws.

• Humans do not scale
• Judge model scores each answer

Known pitfalls include position bias, where the judge prefers whichever answer comes first, and verbosity bias, where longer answers score higher. Judges can also favor outputs from their own model family, and tend to be lenient with fluent but unsupported text. Scores can shift when the judge model is updated, so pin its version.

• Position and verbosity bias
• Self-preference
• Leniency and drift

The fix is calibration. Have a person label a sample of maybe fifty or a hundred answers, then measure how often the judge agrees. Use narrow yes or no questions rather than vague scores out of ten, ask for reasoning before the verdict, and swap answer order when comparing. A judge you have never checked against humans is just an opinion generator.

• Label a human sample
• Measure agreement
• Use narrow rubrics

A metric is only as good as its test set. Start with real questions from users, support tickets, or search logs, since these reflect what people actually ask. For each, record the relevant chunks and a short reference answer. Even fifty carefully chosen questions will teach you more than a thousand invented ones.

• Real user questions
• Labelled chunks and answers

To scale up, have an LLM generate questions from your chunks. The chunk it started from becomes the labelled relevant chunk. But synthetic questions are often too easy, because they copy wording from the chunk, which flatters keyword search. Review a sample, delete vague ones, and rewrite some in the casual style of real users.

• LLM writes questions from chunks
• Human review is required

Next, add the hard cases. Unanswerable questions, where the right behavior is to say the documents do not contain it. Multi-hop questions, which need facts from two different chunks, such as comparing two products. And ambiguous or follow-up queries. If your test set only contains easy lookups, your scores will be high and misleading.

• Unanswerable questions
• Multi-hop questions
• Ambiguous queries

Now the failure catalogue. Use a diagnosis flow and stop at the first failure you find. First, does the answer exist in the corpus at all? Second, was it retrieved? Third, was it ranked and placed well? Fourth, did the model use it correctly? Working in this order stops you from fixing the prompt when the real problem is missing data.

• Check content, retrieval, then generation
• Follow the first failure

Failure one: missing content. The answer simply is not in your index, because the document was never ingested or parsing dropped a table. Fix the data, not the model. Failure two: the content exists but was not retrieved. Recall at k is low, so try better chunking, query rewriting, hybrid search, or a larger candidate pool before reranking.

• Answer not in corpus
• In corpus, but not fetched

Failure three: lost in the middle. Research has shown that models often use information at the start and end of a long context better than information in the middle. So put your best chunks first, and send fewer, better ones. Failure four: wrong chunk. The retrieved text is topically close but lacks the answer, or a split cut the key sentence in half.

• Models favour start and end
• Chunk is close but incomplete

Failure five: outdated documents. Both the old and new policy are indexed, and the retriever returns the old one. Use metadata like dates and versions, filter on them, and remove stale files. Failure six: conflicting sources. Two chunks disagree, and the model picks one silently. Surface the conflict, rank by authority or recency, or have the model mention both.

• Old versions still indexed
• Documents disagree

Failure seven: the prompt ignores context. The right chunk is present, but the model answers from its memory, or blends in unsupported detail. Faithfulness will be low. Tighten the instructions and require citations. Failure eight is the opposite, over-refusal: the evidence is there, but the model says it cannot find the answer. Balance your prompt with both answerable and unanswerable test cases.

• Model answers from memory
• Model refuses despite evidence

Here is the payoff: metrics point to fixes. Low context recall suggests missing content or retrieval problems. Low context precision suggests noise, so rerank or cut k. Low faithfulness with good context points at the prompt or model. High faithfulness but wrong answers points back at stale or conflicting sources. Measure first, then change one thing at a time.

• Low recall: retrieval fix
• Low faithfulness: prompt fix
• Low relevance: question handling

Let's build a mini harness in plain Python. The dataset is a list of dictionaries, each with a question, a set of relevant chunk IDs, and a reference answer. We loop over it, call our retriever, and call our generator. Then we compute metrics per question and average them, keeping every individual result so we can inspect failures.

• Dataset of questions
• Retriever and generator
• Metrics per question

Recall at k is short. Take the first k retrieved IDs, intersect with the relevant set, and divide the size of the intersection by the size of the relevant set. For reciprocal rank, walk the ranked list with enumerate starting at one, and return one over the rank of the first relevant ID, or zero if none appears. Average across all questions.

• Set intersection for recall
• First hit index for MRR

For faithfulness, make two LLM calls through any provider. First, ask the judge to extract the atomic claims from the answer as a list. Second, for each claim, ask whether the context supports it, answering strictly yes or no. The score is the count of yes answers divided by the number of claims. Set temperature to zero and log the judge's reasons.

• Extract claims
• Verify each against context
• Score = supported over total

Now the mistakes. The first is only eyeballing final answers. Five happy-path questions will not reveal a recall problem, and they give no hint about whether retrieval or generation is at fault. Even a small automated suite, run on every change, beats impressions. Look at individual failing cases too, since averages hide patterns.

• Five demo questions are not a test
• No signal on which half failed

The second mistake is tuning on the eval set. If you tweak chunk sizes, prompts, and thresholds until the same fifty questions score well, you are overfitting, exactly as in the ML course's lesson on validation. Split your questions into a development set and a held-out set you rarely touch. Refresh the test set with new real queries over time.

• Overfit prompts and thresholds
• Hold out a test split

To recap: split RAG into retrieval and generation. Score retrieval with recall, precision, MRR, and nDCG. Score generation with faithfulness, relevance, context precision and recall, and correctness, using calibrated judges. Build a test set with real, synthetic, unanswerable, and multi-hop questions, and follow the diagnosis flow through the eight failure modes.

• Separate retrieval from generation
• Metrics, test sets, failure flow

That completes our RAG module. But sometimes retrieval is not enough, and the model itself needs to change its behavior or style. In the next episode we start fine-tuning, beginning with supervised fine-tuning and instruction tuning.

• RAG module complete
• Next: supervised fine-tuning