Chunking Strategies for RAG: How to Split Documents Right

Your RAG bot has the answer sitting in its database, word for word, and still replies that it does not know. Nothing is wrong with the model, the embeddings, or the index. The document was simply cut in the wrong place. Today we fix that, because chunking quietly decides how good your whole system can be.

• Answer exists in the corpus
• Bot says: I don't know
• Culprit: bad chunk boundaries

In the last episode we built the full RAG architecture: embed the question, retrieve the nearest pieces of text, and hand them to the model as context. That diagram had a box labelled chunks, and we skipped over how it gets filled. In this lesson we open that box and look at every common way to fill it.

• Last time: RAG architecture
• Today: the chunks box
• Strategies, trade-offs, code

We will cover fixed-size, sentence, recursive, structure-aware and semantic chunking, plus parent-document retrieval and contextual headers. We will do a real counting example with a fifty-page document, discuss tables, PDFs and OCR, and then implement three chunkers in Python and compare them. We finish with the mistakes that cause most failures.

• Six core strategies
• Worked numeric example
• Python comparison
• Common mistakes

A vector index does not store documents, it stores chunks. Each chunk becomes one vector, and retrieval can only return whole chunks. So the chunk is the retrieval unit, and whatever you put inside it is the most the model can ever receive from that hit. The unit you choose determines what is findable.

• One chunk, one vector
• Retrieval returns whole chunks
• Chunk = unit of findability

Imagine a chunk that covers three topics: refunds, shipping and warranty. Its embedding is a blend of all three, an average that sits between topics and matches none of them sharply. A question about shipping then scores only moderately against it. Big chunks dilute meaning, and they also waste context tokens on irrelevant text.

• Mixed topics blur the vector
• Weak similarity to any query
• Wasted prompt tokens

Now the opposite. Cut so small that a chunk reads, it must be returned within thirty days. Returned what? Within which policy? The pronoun lost its noun, and the chunk is precise but useless. Tiny chunks lose context, so the model gets a fragment it cannot interpret.

• Pronouns lose their referents
• Precise but uninformative
• Fragments confuse the model

The simplest strategy is fixed-size chunking. Pick a size, say five hundred tokens, and slice the text every five hundred tokens regardless of content. It is fast, predictable, and trivial to implement. Its weakness is that it is blind to meaning, so cuts land wherever the counter happens to hit the limit.

• Cut every N tokens
• Simple and predictable
• Blind to meaning

To soften bad cuts, we add overlap. With chunk size five hundred and overlap fifty, each new chunk starts fifty tokens before the previous one ended. A sentence cut at one boundary then appears whole in the neighbouring chunk. Overlap is cheap insurance, but it duplicates text, which costs storage and embedding compute.

• Overlap repeats boundary text
• Chunk 500, overlap 50
• Cost: duplicated tokens

Take a fifty-page document at roughly five hundred tokens per page, so twenty-five thousand tokens. With no overlap, five hundred token chunks give exactly fifty chunks. With overlap fifty, each step advances four hundred fifty tokens. We need the first chunk plus enough steps to cover the remaining twenty-four thousand five hundred tokens.

• 50 pages, about 25,000 tokens
• Stride = 500 minus 50 = 450
• Remaining after first: 24,500

Twenty-four thousand five hundred divided by four hundred fifty is about fifty-four point four, and we round up to fifty-five more steps. Add the first chunk and we get fifty-six chunks. That is twenty-eight thousand tokens to embed, about twelve percent more than the original twenty-five thousand. The formula is ceiling of total minus size over stride, plus one.

• 24,500 / 450 ≈ 54.4, round up to 55
• 55 + 1 = 56 chunks
• 28,000 tokens embedded, about 12% extra

A smarter baseline respects natural boundaries. Split into sentences or paragraphs first, then pack whole sentences into a chunk until adding another would exceed the limit. Now no sentence is ever cut in half. Chunk sizes vary a little, but each chunk reads as coherent text, which embedding models handle far better.

• Split at sentence boundaries
• Pack sentences up to the limit
• Chunks stay readable

Recursive splitting is the popular default in many libraries. You give it an ordered list of separators: double newline, single newline, space, then empty string. It tries the biggest separator first. If a piece is still too large, it recurses into that piece using the next separator. Paragraphs are kept whole when they fit.

• Separators: paragraph, line, space, char
• Try the coarsest first
• Recurse only on oversized pieces

The result is chunks that follow the document's own shape as closely as the size limit allows. Most cuts land at paragraph breaks, some at line breaks, and word-level cuts happen only as a last resort. It needs no model and no tuning beyond size and overlap, so it is a strong first choice for plain prose.

• Respects natural breaks
• No model needed
• Strong default for prose

Many documents carry explicit structure. In Markdown, headings mark sections, and a section is usually a self-contained idea. A heading-aware chunker splits at headings, keeps each section together, and attaches the heading path, like Billing then Refunds, as metadata. Only oversized sections get split further, using a recursive pass inside them.

• Split on heading boundaries
• Keep the heading path as metadata
• Subdivide only long sections

HTML offers the same signal through tags: headings, lists, tables and paragraphs. For source code, splitting by lines is risky because it can cut a function in half. Better is parsing code into an abstract syntax tree and chunking by function or class, so each chunk is a complete, meaningful unit with its name and signature.

• HTML tags reveal structure
• Code: chunk by function or class
• Parse with an AST

When structure exists, use it. Authors already told you where ideas begin and end, and ignoring that to cut by token count throws information away. The practical rule is to parse structure first, then enforce size limits second. Size limits protect the embedding model, but structure protects meaning.

• Structure first, size second
• Authors mark idea boundaries
• Size limits protect the model

Semantic chunking lets the content choose the boundaries. Embed each sentence, then compare neighbouring sentences with cosine similarity. While similarity stays high, the topic continues. When it drops sharply, you have likely crossed a topic change, so you start a new chunk there. It is the same cosine similarity from our embeddings lessons, used as a cutting tool.

• Embed every sentence
• Compare neighbours by cosine
• Cut where similarity drops

Suppose six sentences give neighbour similarities of zero point eight two, zero point eight, zero point four one, zero point seven eight, and zero point seven nine. With a threshold of zero point six, only the third value falls below it. So we cut once, between sentences three and four, giving two chunks. Thresholds are often set from a percentile of the observed drops.

• Similarities: 0.82, 0.80, 0.41, 0.78, 0.79
• Threshold: 0.6
• One cut after sentence 3

Semantic chunking costs an embedding call per sentence at indexing time, and the results can be unstable, with chunk sizes ranging wildly. Studies and practitioner reports on its benefit over simple recursive splitting are mixed, so do not assume it wins. Test it on your own data, and add minimum and maximum size limits.

• Extra indexing cost
• Unpredictable chunk sizes
• Benefit is data-dependent

Here is a trick that resolves the size dilemma. Small chunks match queries precisely, but large chunks give the model context. So index small child chunks, say two hundred tokens, but link each to a larger parent, like its full section. Search finds the child, and you send the parent to the model. This is parent-document, or small-to-big, retrieval.

• Embed small children
• Return the larger parent
• Precision plus context

Another fix for lost context is to enrich each chunk before embedding it. Prepend the document title, the section path, and sometimes a one or two sentence summary of where the chunk sits. Then it returned within thirty days becomes part of the Acme Electronics refund policy, and it embeds far more usefully. Generating the summary with an LLM adds indexing cost, so measure the gain.

• Prepend title and section path
• Optional LLM-written context line
• Embed the enriched text

Chunk size is a dial with four forces pulling on it. Smaller chunks raise precision but can hurt recall of answers spread across passages. Larger chunks carry more context but dilute embeddings. The embedding model has a token limit, and the language model has a context budget that you pay for on every call.

• Precision versus recall
• Embedding model token limit
• LLM context cost

Every embedding model has a maximum input length. Anything beyond that limit is truncated by many systems, so the tail of an oversized chunk is never represented in its vector. Always check your model's limit and count tokens with that model's own tokenizer, because characters and words are only rough guides.

• Check the model's max tokens
• Overflow is truncated
• Count with the real tokenizer

Now the cost side. If you retrieve the top five chunks of five hundred tokens, that is twenty-five hundred tokens of context per question. Move to two-thousand-token chunks and the same top five becomes ten thousand tokens. At thousands of questions a day, that difference shows up in your bill and in latency, as we will study in the latency and cost lesson.

• 5 x 500 = 2,500 tokens
• 5 x 2,000 = 10,000 tokens
• 4x more cost per question

There is no universal best size. A reasonable starting range for prose is a few hundred tokens, often between two hundred fifty and eight hundred, with ten to fifteen percent overlap. Treat that as a hypothesis, not a rule. Factual lookup questions favour smaller chunks, while questions needing explanation favour larger ones.

• Start: a few hundred tokens
• Overlap about 10 to 15 percent
• Tune on real questions

Back to our fifty-page example. Overlap of fifty gave fifty-six chunks. Raise overlap to two hundred and the stride drops to three hundred, so we need ceiling of twenty-four thousand five hundred over three hundred, which is eighty-two, plus one, giving eighty-three chunks. That is forty-one thousand five hundred tokens embedded. Heavy overlap also returns near-duplicate results.

• Overlap 200, stride 300
• 82 + 1 = 83 chunks
• 41,500 tokens, near-duplicate hits

Tables are where chunkers fail loudly. A row cut away from its header row is just numbers with no meaning. The fix is to keep tables whole when they fit, or split by rows and repeat the header in every piece. Often it helps to convert the table to Markdown or add a sentence describing what it shows.

• Never cut a table mid-row
• Repeat header row in each piece
• Add a short description

A PDF is a drawing format, not a text format. Extraction can scramble multi-column layouts, interleaving left and right columns, and headers, footers and page numbers leak into the body text. Clean those out, use a layout-aware parser, and inspect the extracted text by eye before you chunk. Garbage in, garbage chunks.

• PDF stores positions, not flow
• Columns can interleave
• Strip headers, footers, page numbers

Scanned documents need optical character recognition, and OCR makes errors: the letter l becomes the digit one, and broken words appear at line ends. Those errors change tokens and weaken matching. Spot-check samples, repair hyphenated line breaks, record confidence where the tool provides it, and always keep page numbers as metadata so answers can cite the source page.

• OCR confuses similar glyphs
• Fix hyphenated line breaks
• Keep page numbers as metadata

Now let us code. Use plain Python and a small Markdown sample with a title, two headed sections, and a few paragraphs about a refund policy. To keep it simple, we approximate tokens by words, though in production you should use the real tokenizer. We will write three functions and compare how each one cuts the same text.

• Small Markdown sample
• Tokens approximated by words
• Three chunkers, one text

The fixed chunker converts text to a list of words, then loops with a step equal to size minus overlap. At each position it takes a slice of size words and joins them back into a string. That is only a few lines, and it matches our stride formula exactly. Note that it will happily cut in the middle of a sentence.

• words = text.split()
• step = size - overlap
• slice words[i : i + size]

The recursive chunker takes a list of separators. If the text fits, return it. Otherwise split on the first separator, then pack the pieces greedily, and recurse into any piece still too long using the next separator. The heading-aware chunker uses a regular expression to split before each line starting with hash marks, tracks the current heading path, and stores it as metadata.

• Recursive: separators list, greedy pack
• Heading-aware: regex on # lines
• Store heading path as metadata

Run all three on the sample and print each chunk. The fixed chunker produces equal sizes but breaks a sentence about the thirty-day window across two chunks. The recursive chunker mostly ends at paragraph breaks. The heading-aware chunker gives one chunk per section, each tagged with its path, though one section is much longer than the others.

• Fixed: equal size, broken sentence
• Recursive: paragraph-aligned
• Heading-aware: tagged, uneven size

Here are the mistakes to avoid. Splitting mid-sentence or mid-table. Throwing away metadata like source, page, section and date, which you need for filtering and citations. Using one chunk size for every document type, when a legal contract, a chat log and a code repository each deserve their own strategy. And never looking at your actual chunks.

• Mid-sentence or mid-table cuts
• Losing metadata
• One size for all documents
• Never inspecting chunks

How do you know a strategy is better? Write a set of real questions and mark which source passage answers each one. Build the index with each chunker, retrieve the top five for every question, and count how often the right passage appears. That fraction is the hit rate at five. Change only one thing at a time, and the winner on your data is your answer.

• Questions with known source passages
• Retrieve top 5 per chunker
• Hit rate = hits / questions

Put it together. Parse structure first. Split long sections recursively with modest overlap. Attach heading paths and source metadata. Consider small-to-big retrieval for context. Then measure hit rate on real questions and iterate. Complete evaluation of retrieval and generation comes in a later episode, but even this quick check beats guessing.

• Parse, split, attach metadata
• Consider small-to-big
• Measure, then iterate

To recap: the chunk is the retrieval unit, so it limits what can be found. Fixed-size is simple, sentence and recursive splitting respect text, structure-aware uses headings and syntax trees, and semantic chunking cuts at topic shifts. Parent-document retrieval and contextual headers restore context. A fifty-page document at five hundred tokens with fifty overlap gives fifty-six chunks. Tables, PDFs and OCR need care.

• Chunk = retrieval unit
• Strategies and trade-offs
• 50 pages gives 56 chunks

Good chunks are only half the story, because the user's question is often vague or phrased nothing like your documents. In the next episode we cover retrieval and query rewriting: how to turn messy questions into queries that find the right chunks.

• Next: retrieval and query rewriting
• Messy questions to good queries
• Subscribe for episode 17