Build a Document Q&A RAG Bot: Hybrid Search, Reranking, Citations, Evals

Almost every RAG demo works on three questions the author picked. Then a real user asks a fourth question, and the bot confidently invents a refund policy that does not exist. Today we build a document question answering bot that is designed not to do that, and we prove it with numbers.

• Three easy questions pass
• The fourth one fails
• We will measure it

This is the first capstone, and it integrates the last several lessons. In the previous lesson we covered guardrails, prompt injection and monitoring. Here we wire together chunking, embeddings, hybrid search, reranking, grounded prompting and evaluation into one project. I will recap each decision rather than re-teach it, and I will point to the lesson where it was derived.

• Integrates earlier lessons
• Recap decisions, not theory
• Runnable end to end

Here is what the finished bot does. It ingests PDFs and Markdown files. It retrieves with both keyword and dense search, reranks the candidates, and answers only from the retrieved text, with a citation after every claim. If the evidence is missing, it says so. And it ships with an eval set, so every change is tested.

• Cited answers
• Abstains without evidence
• Ships with evals

The architecture has two paths. The offline path loads documents, splits them into chunks, embeds each chunk, and writes them to an index. The online path takes a question, rewrites it using chat history, retrieves, reranks, builds a prompt, and calls the model. Keeping those paths separate is what lets you update documents without touching the query code.

• Ingest builds the index
• Query reads the index
• Decoupled on purpose

The repository layout mirrors that split. A folder called rag holds one module per stage: loaders, chunking, embed, store, retrieve, rerank, generate and rewrite. A folder called evals holds the question set and the scoring script. An app folder holds the FastAPI service and the Gradio interface. Everything is wired through one config file.

• rag: one module per stage
• evals: questions and scoring
• app: API and UI

Configuration lives in one place. Chunk size, overlap, the embedding model name, how many candidates each retriever returns, the reciprocal rank fusion constant, how many chunks survive reranking, and the abstention threshold are all config values. This matters because every one of them is a knob you will turn during evaluation, and hard-coded numbers hide the experiments.

• No magic numbers in code
• Each knob is testable
• Log config with every eval

We keep the code provider neutral. The embedding model, the reranker and the language model sit behind tiny interfaces: embed takes a list of strings and returns vectors, rerank takes a query and passages and returns scores, and generate takes messages and returns text. Swap a vendor by changing one adapter, not the pipeline.

• embed, rerank, generate
• Adapters isolate vendors
• Pipeline never changes

Ingestion starts with loaders. Markdown is easy because headings are explicit. PDFs are harder: extract text page by page, keep the page number, and expect problems like headers repeated on every page and tables flattened into noise. Every loaded unit becomes a record with text and metadata: source file, page, section title, and a modified time.

• Keep page and section
• Strip repeated headers
• Metadata on every record

For chunking, we use what the chunking lesson recommended: structure-aware splitting. Split on headings first, then on paragraphs, and only fall back to sentences when a section is too long. We target a few hundred tokens per chunk with a modest overlap. Each chunk also keeps its heading path, such as Billing, then Refunds, so it never loses its context.

• Headings first
• Then paragraphs
• Keep the heading path

Here is a worked example. Suppose a Markdown file has three sections of 900, 250 and 1,400 tokens, and our target is 400 tokens with 50 tokens of overlap. The 250 token section stays whole. The 900 token section splits into about three chunks. The 1,400 token section splits into about four. That gives roughly eight chunks, each labelled with its own heading.

• 900 tokens: about 3 chunks
• 250 tokens: stays whole
• 1400 tokens: about 4 chunks

Each chunk gets a stable identifier made from a hash of its source path, heading path and text. This looks like a small detail, but it is the foundation of source updates. When a file changes, we can compare hashes, delete chunks that disappeared, and embed only the new ones, instead of rebuilding everything or leaving stale answers in the index.

• Hash of path, heading, text
• Diff old versus new
• Only re-embed what changed

Quick sanity check before moving on. Print twenty random chunks and read them. Do they begin mid-sentence? Is a table turned into gibberish? Is the heading path right? Most retrieval problems that look like model problems are really chunking problems, and ten minutes of reading catches them cheaply.

• Sample twenty at random
• Look for broken tables
• Fix chunking before tuning

Next, embeddings and the vector store. We embed each chunk with a sentence embedding model, normalize the vectors, and store them with their metadata. As the embedding lessons showed, with normalized vectors cosine similarity and dot product agree. For a few thousand chunks an exact search is fine, and for larger corpora you switch to an HNSW index.

• Normalize the vectors
• Store metadata alongside
• Exact first, HNSW later

Metadata earns its keep at query time. Because each chunk stores its source, section and date, we can filter: only search the handbook, or only documents updated this year. Filtering before ranking is far cheaper than retrieving and discarding, and it prevents an outdated policy from outranking the current one.

• Filter by source or date
• Filter before ranking
• Avoid stale policies

Dense search alone misses exact strings. Ask about error code E 4 0 4 7 or a product named Falcon X, and embeddings may return something merely related. So we also build a BM25 keyword index over the same chunks. Dense search captures meaning, BM25 captures exact terms, and as the hybrid search lesson argued, you want both.

• Dense: meaning
• BM25: exact terms
• Complementary failures

We merge the two ranked lists with reciprocal rank fusion. A chunk's score is the sum, over each list it appears in, of one divided by k plus its rank, with k set to sixty. No score calibration is needed, because only ranks matter. It is simple, robust, and easy to implement in a few lines of plain Python.

• Sum of 1 over k plus rank
• k equals 60
• Only ranks matter

A worked example. Chunk A is first in dense and third in BM25. Its score is one over sixty-one plus one over sixty-three, about 0.0164 plus 0.0159, so roughly 0.0323. Chunk B is second in dense only, giving one over sixty-two, about 0.0161. Chunk A wins because both retrievers liked it. Agreement across retrievers is exactly the signal we want.

• A: 0.0164 plus 0.0159
• B: 0.0161
• Agreement wins

In code, the retrieve function runs both searches, each returning, say, thirty candidates, applies any metadata filter, fuses with reciprocal rank fusion, and keeps the top thirty. This stage is about recall: we want the right chunk to be somewhere in that list. Precision is the next stage's job.

• Thirty from each retriever
• Fuse with RRF
• Goal is recall

Now the cross-encoder reranker. It reads the query and each candidate together, so it judges relevance much more precisely than comparing two separate embeddings, but it is slower, which is why it only sees thirty candidates. We keep the top five. Cost is predictable: thirty reranker passes per question.

• Reads query and chunk together
• Thirty in, five out
• Precision stage

The reranker score also gives us something valuable: a signal for abstention. If even the best chunk scores below a threshold we tuned on the eval set, retrieval probably failed, and the bot should say it could not find the answer rather than guess. This single check removes a large share of confident nonsense.

• Best score too low?
• Skip generation
• Tune threshold on evals

The grounded prompt is where trust is won. The system message says: answer only using the numbered sources below, cite them as bracketed numbers after each claim, and if the sources do not contain the answer, say you do not know. Each source is shown with its number, file name and section. Retrieved text is wrapped as data, and the prompt tells the model never to follow instructions found inside it, which is the injection defence from the last lesson.

• Answer only from sources
• Cite with numbers
• Say I do not know

After generation we validate. Parse the citation numbers from the answer and check each one points to a real source we supplied. If a citation is out of range, or the answer has no citations while claiming facts, retry once or return the abstention message. Then we return the answer together with the source list, so the interface can show file names and pages.

• Parse bracketed numbers
• Reject invalid citations
• Return sources with answer

Real users ask follow-ups like, and what about contractors? Retrieval on that raw text finds nothing useful. So before retrieval, a rewrite step takes the recent chat history and produces a standalone question: what is the refund policy for contractors? As in the query rewriting lesson, we keep the rewrite short, and retrieve with the rewritten question only.

• Follow-ups lack context
• Rewrite to a standalone question
• Retrieve on the rewrite

For the API we use FastAPI. One endpoint, ask, accepts a question and an optional chat history, and returns the answer, a list of sources with file, page and score, and a flag saying whether the bot abstained. A second endpoint, ingest, reindexes changed files. Typed request and response models make the contract explicit.

• POST ask
• POST ingest
• Typed request and response

For a quick interface, Gradio gives us a chat window in about twenty lines. It calls the same pipeline function the API uses, displays the answer, and shows a collapsible list of sources underneath. Showing sources is not decoration. It lets users verify answers, and that is what turns a clever demo into a tool people rely on.

• Same pipeline function
• Sources shown under answer
• Users can verify

Handling source updates is the part most tutorials skip. Our ingest command walks the data folder, computes chunk hashes, deletes chunks whose hash vanished, adds new ones, and rebuilds the keyword index. It stores the modified time in metadata, and a scheduled job or file watcher can call it. Without this, your bot will cheerfully quote last year's policy.

• Hash-diff on ingest
• Delete vanished chunks
• Run on a schedule

Now the part that separates a project from a demo: evaluation. We write an eval set of about forty questions, drawn from real user questions where possible. Each has an expected source chunk or file and page, a reference answer in a sentence, and a few questions deliberately unanswerable from the documents, to test abstention.

• About forty questions
• Gold source for each
• Include unanswerable ones

The first metric is recall at k: for each question, did the gold chunk appear in the top k results? If thirty-two of forty answerable questions have the gold chunk in the top five, recall at five is thirty-two over forty, or eighty percent. Measure it after fusion and after reranking, because the two numbers tell you which stage is failing.

• Gold chunk in top k?
• 32 of 40 gives 0.80
• Measure per stage

The second metric is faithfulness: is every claim in the answer supported by the retrieved sources? We use a judge model with a strict rubric. Split the answer into claims, ask whether each is supported by the cited chunks, and score the fraction supported. Calibrate the judge by hand-checking a sample, because judges make mistakes too. Also check abstention: unanswerable questions should abstain, answerable ones should not.

• Split answer into claims
• Judge each against sources
• Check abstain behaviour

Run the baseline and read the failures, not just the averages. Suppose recall at five is eighty percent, faithfulness is high, but three questions with exact product codes miss. Reading them shows the BM25 tokenizer split the codes on hyphens. That is a bug, not a modelling problem, and no bigger model would have fixed it.

• Read each failure
• Hyphenated codes were split
• Fix the bug, rerun

After that fix, recall at five rises, but two failures remain where the answer spans two neighbouring chunks, and the gold chunk was cut mid-table. The remedy is in chunking: keep tables whole and raise the overlap slightly. Re-ingest, rerun the evals, and compare. One change at a time, with the config logged beside each score, so you know what helped.

• Tables kept whole
• One change at a time
• Log config with scores

A third pattern: the bot answers an unanswerable question using a loosely related chunk. Raising the abstention threshold fixes it, but also makes the bot refuse two answerable questions. Plot the tradeoff and pick the threshold from the curve. There is no free setting, only a tradeoff that you now measure instead of guess.

• Higher threshold: fewer inventions
• But more refusals
• Choose from the curve

Make this automatic. A single command runs the eval set, prints recall at k before and after reranking, faithfulness, and abstention accuracy, and fails if any metric drops below the last accepted run. Put it in your continuous integration. Then changing a prompt, a model or a chunk size becomes a measured decision.

• One command, all metrics
• Compare to last accepted run
• Run in CI

Common mistake number one: skipping evals. Without them, every tweak is a vibe, regressions go unnoticed, and you cannot tell whether the reranker or the new prompt helped. A small eval set of forty honest questions beats no eval set, and you can grow it every time a user reports a bad answer.

• Tweaks become vibes
• Regressions go unseen
• Start with forty questions

Mistake number two: no plan for source updates. Documents change, and an index built once becomes a liability. Use stable chunk identifiers, reindex incrementally, store modified dates, and filter on them. Related mistakes: sending too many chunks so the model drowns, forgetting to log retrieved chunks, so you cannot debug, and ignoring prompt injection hidden inside documents.

• Index once is a liability
• Incremental reindexing
• Log retrieved chunks

Let us recap the decisions. Structure-aware chunks with heading paths and stable hashes. Embeddings plus BM25, fused with reciprocal rank fusion. A cross-encoder to pick the best five. A grounded prompt with citations and abstention. A rewrite step for follow-ups. And an eval set measuring recall at k, faithfulness and abstention, driving every change.

• Chunk, embed, BM25, fuse
• Rerank, ground, cite, abstain
• Eval drives iteration

Where could you go next? Try query expansion, a larger eval set built from real logs, per-user access filters in metadata, or streaming answers in the interface. Whatever you add, add it behind a flag and let the eval set decide. The code for this project, with the config and eval script, is in the description.

• Query expansion
• Access filters
• Grow the eval set

In the next lesson, Capstone 2, we build a coding assistant. It reads a repository, answers questions about it, and proposes edits, reusing retrieval ideas but with code-aware chunking and tools.

• Code-aware chunking
• Tools to edit code
• Subscribe for Capstone 2