RAG Explained: How to Ground LLMs in Your Own Data

Imagine you ask a language model, what is our company's refund policy? It answers fluently, with total confidence, and it is completely made up. The model has never seen your policy. It is guessing, and it sounds exactly as sure as when it is right.

• Model never saw your documents
• Confident but wrong answers

In the last lesson we looked at vector databases in practice: storing embeddings and searching them fast. Today we put everything together. This is the RAG architecture, the pattern that grounds a language model in your data, and it is behind most real-world LLM products.

• Lesson 14: vector databases
• Today: the full system

By the end of this video you will understand every stage of the pipeline, the probability idea behind it, a prompt template that forces grounded answers, a worked example with real numbers, and a minimal implementation in about sixty lines of Python. We will also cover the three mistakes that sink most first attempts.

• Pipeline, math, prompt, code
• Mistakes to avoid

Here is the intuition. A plain language model takes a closed-book exam. Everything it knows must come from memory, stored in its weights during training. RAG turns it into an open-book exam: before answering, the system finds the relevant pages and hands them to the model to read.

• Closed book: memory only
• Open book: look it up

Recall from lesson six why models hallucinate: they predict plausible text, not verified facts. Add the knowledge cutoff, since the model knows nothing after training ended, and private data it never saw at all. Retrieval attacks all three, because the facts arrive fresh, in the prompt, at question time.

• Hallucination
• Knowledge cutoff
• Private data

Why not just fine-tune the model on your documents? Fine-tuning is good at changing style and behavior, but it is a slow and unreliable way to inject facts, and when a document changes you must train again. With RAG you simply update the index. We will compare these options properly in lesson twenty-six.

• Fine-tuning changes behavior
• RAG supplies facts
• Facts change often

A RAG system is really two pipelines. The ingestion pipeline runs offline, whenever your documents change, and prepares your knowledge for search. The query pipeline runs online, on every user question. Let us build the ingestion side first, one stage at a time.

• Ingest: offline, once
• Query: online, every question

First, load. Your knowledge lives in PDFs, wiki pages, tickets, and web pages. Then clean: remove navigation menus, repeated headers, and broken characters, because garbage text becomes garbage embeddings. Crucially, keep metadata with every document, such as its source, title, date, and who is allowed to see it.

• PDFs, wikis, tickets, HTML
• Strip boilerplate
• Keep metadata

Next, chunk. A whole handbook is far too big to embed as one vector, and one vector for many topics blurs them all together. So we split the text into passages, often a few hundred tokens each, sometimes with a little overlap. The right way to chunk is subtle, and it gets the whole next lesson.

• Documents are too long
• Split into passages
• Overlap helps

Then embed. Each chunk goes through an embedding model, as in lessons eleven and twelve, and comes out as a vector where similar meanings sit close together. We store the vector, the original text, and the metadata, and build an approximate nearest-neighbor index over the vectors, as in lesson thirteen. Ingestion is done.

• Chunk to vector
• Store vector plus text
• Build ANN index

Now a user asks a question. Step one of the query pipeline: embed the question with the same embedding model you used for the chunks. This detail matters. Vectors from two different models live in different spaces, so comparing them is meaningless.

• Same embedding model
• Same vector space

Step two: retrieve. Ask the index for the k nearest chunks, where k is a small number like three to eight. The index returns the chunk texts, their similarity scores, and their metadata. These are the pages we are about to hand to the model.

• Nearest chunks by similarity
• k is usually small
• Return text and metadata

Step three: assemble the prompt. We combine instructions, the retrieved chunks, and the user's question into one message. Step four: generate. The language model reads that context and writes an answer. Notice the model's weights never changed. We only changed what it gets to read.

• Instructions + context + question
• Model reads, then answers

Step five: cite. We label each chunk in the prompt, like source one, source two, and ask the model to reference those labels in its answer. The application then turns labels into links. Citations let users verify the answer, and they let you debug the system when something goes wrong.

• Label each chunk
• Model cites labels
• User can verify

Now the math, as intuition. The original RAG paper from 2020 describes the answer probability like this. The probability of answer y given question x equals the sum over documents d of the probability of retrieving d given x, times the probability of generating y given both x and d.

• p(y|x) = Σ p(d|x) p(y|x,d)
• Retriever times generator

Read it as a weighted vote. Each document gets a weight from the retriever. For each document, the generator says how likely the answer is if it read that document. We blend them. Suppose document A has weight zero point seven and supports our answer with probability zero point nine, while document B has weight zero point three and supports it with probability zero point two. The total is zero point six three plus zero point zero six, which is zero point six nine.

• Weight each document
• Answer given that document
• Blend the results

A caveat: this is intuition, not what most production systems compute. Summing over every document is too expensive, so we keep only the top k, which approximates the sum since the other weights are tiny. Most applications then simply paste those k chunks into one prompt and let the model weigh them itself through attention.

• Exact sum is expensive
• Top-k approximates the sum
• Concatenate into the prompt

The prompt template does a lot of work. A good one says: you are a support assistant. Answer using only the context below. Cite sources by their label. If the context does not contain the answer, say you don't know. That last instruction gives the model a safe exit instead of an invented answer.

• Answer only from context
• Cite sources
• Allow 'I don't know'

Each line earns its place. Answer only from the context discourages the model from mixing in memorized guesses. Citation labels force it to point at evidence. The refusal rule matters most when retrieval fails, because then a grounded model says it doesn't know, rather than making something up. Remember, this reduces hallucination. It does not eliminate it.

• Only from context: limits guessing
• Citations: verifiability
• Refusal: honest gaps

Format matters too. Separate the chunks with clear delimiters and number them so citations are unambiguous. Put the instructions first and the question last, close to where generation starts. Models can lose track of material buried in the middle of a long prompt, which is one more reason to keep the context short.

• Clear separators
• Number the chunks
• Question near the end

Let us run a full example. Our tiny knowledge base has three chunks. Document A says, items can be returned within thirty days of delivery for a full refund. Document B says, standard shipping takes three to five business days. Document C says, electronics carry a one year warranty. The user asks: how long do I have to return an item?

• A: Returns policy
• B: Shipping times
• C: Warranty terms

To keep the numbers simple, imagine two-dimensional unit vectors. The question is zero point six, zero point eight. Document A is zero point eight, zero point six. The dot product is zero point four eight plus zero point four eight, which is zero point nine six. Document C is zero, one, giving zero point eight. Document B is one, zero, giving zero point six. Real vectors have hundreds of dimensions, but the arithmetic is identical.

• Unit vectors: dot = cosine
• A: 0.96
• C: 0.80
• B: 0.60

Rank by score: A at zero point nine six, C at zero point eight, B at zero point six. With k equal to two, we keep A and C and drop B. Notice something important. C, the warranty, is not really relevant, but it scored high anyway. Similarity means related, not necessarily useful. Later lessons attack exactly this with reranking.

• Rank by score
• Take k = 2
• B is dropped

Now assemble. The system rules come first. Then context: source one, the returns policy text, and source two, the warranty text. Then the question. The model reads this and should answer: you can return an item within thirty days of delivery for a full refund, citing source one. The warranty chunk is ignored, because the answer lives in the other.

• Rules, then context, then question

Now ask something the knowledge base cannot answer, like, can I return a gift card? The top chunks will still be returned, because nearest neighbors always exist. They are just not relevant. A grounded prompt lets the model reply that the context does not say. Without that rule, it would happily improvise a policy.

• Ask about gift cards
• No relevant chunk
• Good system: I don't know

Time for code. The whole system is four small functions. An embed function returns a normalized vector. A retrieve function does numpy search. A build prompt function formats the template. And call llm is a provider-neutral stub: it takes a string and returns a string, so you can plug in any model API you like.

• embed()
• retrieve()
• build_prompt()
• call_llm()

Ingestion is a few lines. Embed every chunk once and stack the vectors into a matrix of shape number of chunks by dimension, with each row normalized to length one. That matrix is our entire vector database. For a few thousand chunks, brute-force search over it is fast enough, and exact.

• Embed all chunks once
• Stack into a matrix
• Normalize rows

To retrieve, embed the question into a vector q, then compute scores equals the matrix E times q. That single matrix-vector product gives every cosine similarity at once. Use argsort to take the top k indices. Then build the prompt by numbering each chunk, adding the rules, and appending the question.

• scores = E @ q
• argsort, take top k
• Format with numbered labels

The final answer function chains it all: retrieve, build the prompt, call the model, and return both the text and the source labels. That is roughly sixty lines with comments, and it is a real, working RAG system. Everything else in this course section is making each of these steps better, faster, or safer.

• answer(question)
• Return text plus sources
• Roughly 60 lines total

What we built is naive RAG: embed, retrieve, stuff, generate. It works surprisingly well on demos and then struggles on real data. Chunks cut sentences in half. Users ask vague questions. Similar is not the same as relevant. And without measurement you cannot tell whether a change helped.

• Bad chunks
• Bad queries
• Noisy ranking
• Weak evaluation

Here is the map of the coming lessons, without teaching them yet. Chunking improves what we store. Retrieval and query rewriting improve what we ask. Reranking with cross-encoders improves the ordering. Hybrid search combines keyword and dense search. And evaluation tells us whether any of it actually helps, and why systems fail.

• 16 Chunking
• 17 Query rewriting
• 18 Reranking
• 19 Hybrid search
• 20 Evaluation

Where is this used? Customer support bots answering from help-center articles and past tickets. Internal assistants that search wikis, policies, and engineering docs. Assistants over legal or product documentation. In each case the pattern is the same: the knowledge changes often, answers must be traceable, and the data is private.

• Customer support bots
• Internal knowledge search
• Docs and code assistants

Mistake number one: dumping too many chunks. It feels safe to retrieve twenty passages, but irrelevant text distracts the model, can bury the right passage in the middle, and costs more tokens and time on every request. Start with a small k, measure, and increase only if it helps.

• More context is not better
• Noise distracts the model
• Higher cost and latency

Mistake number two: no citations. Without them, users cannot verify an answer and you cannot tell whether a wrong answer came from bad retrieval or bad generation. Always log the retrieved chunks with each response. That single habit turns mysterious failures into diagnosable ones.

• Unverifiable answers
• Impossible to debug
• Trust erodes

Mistake number three, and the most dangerous: ignoring access control. If your index mixes documents from different teams, a clever question can retrieve something the user should never see, and the model will helpfully summarize it. Store permissions as metadata, and filter during retrieval, before anything reaches the prompt.

• Retrieval can leak documents
• Filter before generation
• Store permissions as metadata

Let us recap. RAG makes the model take an open-book exam. Ingestion loads, cleans, chunks, embeds, and indexes. Querying embeds, retrieves, assembles a grounded prompt, generates, and cites. The formula is a retriever weight times a generator probability, approximated by pasting the top k chunks. And watch for too many chunks, missing citations, and leaky permissions.

• Ingest, then query
• Ground with context and citations
• Measure and improve

Next lesson is chunking strategies. How big should a chunk be, where should you split, and why can a tiny change here swing your whole system's quality?

• How big should a chunk be?
• Where to split?
• Subscribe for lesson 16