Hybrid Search Explained: BM25 + Dense Vectors with Reciprocal Rank Fusion

You build a support bot with beautiful embeddings. A user pastes the exact error code E R R underscore five oh four, and your vector search returns three articles about general network trouble. The one page that contains that exact code is nowhere in the top ten. Today we fix that.

• Query: ERR_504_GATEWAY
• Dense search returns vague results

This is lesson nineteen. Last time we reranked candidates with a cross-encoder, which assumes the right document is already among the candidates. Here we make sure it is, by combining keyword search, BM25, with dense embeddings, and merging them using reciprocal rank fusion.

• BM25 from scratch
• Dense retrieval
• Fusion with RRF

Keyword search asks whether the same words appear. It is superb for product IDs, function names, people, and error codes. Embeddings ask whether the meaning is close. They are superb when the user says car and the document says automobile. Each one is strong exactly where the other is blind.

• Keywords: IDs, names, codes
• Embeddings: paraphrase, intent

Another way to see it: a keyword retriever represents a document as a huge sparse vector, one slot per vocabulary word, almost all zeros. A dense retriever uses a few hundred real numbers where every dimension carries a little of the meaning. Same goal, opposite geometry.

• Sparse: one slot per word
• Dense: few hundred real numbers

Start with the simplest idea: score a document by how many times the query words appear. This breaks quickly. Common words like the dominate the count, and a long document wins just by being long. BM25 fixes this with three ingredients: inverse document frequency, saturating term frequency, and length normalization.

• Raw count favors long documents
• Rare words matter more

Let N be the number of documents and n the number containing the term. A word in every document tells us nothing; a word in one document is a strong signal. The common smoothed form is the natural log of one plus N minus n plus point five, over n plus point five. Rarer terms get a bigger weight.

• N documents, n contain the term
• IDF = ln(1 + (N - n + 0.5)/(n + 0.5))

Next, term frequency. If a document mentions gateway once, that is evidence. Ten times is more evidence, but not ten times more. BM25 uses tf times k1 plus one, over tf plus k1. This curve rises fast and flattens toward a ceiling of k1 plus one. With k1 equal to one point two, the ceiling is two point two.

• Second mention helps less
• Ceiling at k1 + 1

Third, length. A long document mentions many words by chance, so a match there is weaker evidence. We replace k1 with k1 times one minus b plus b times document length over average length. When b is zero, length is ignored. When b is one, full normalization. Point seven five is the common default.

• Long docs mention words by chance
• b controls the penalty

Put it together. The score of document d for query q is the sum, over each query term, of IDF times tf times k1 plus one, divided by tf plus K, where K is k1 times the length-normalization factor. No training, no GPU, just counting. Let us run it on real numbers.

• Sum over query terms
• IDF × saturated, normalized TF

Our corpus has three documents. D1 reads: gateway error five oh four gateway timeout, five tokens after dropping nothing. D2 is: error error error in the server logs, seven tokens. D3 is: the gateway routes traffic to servers, six tokens. Average length is five plus seven plus six, divided by three, which is six.

• D1: gateway error 504 gateway timeout
• D2: error error error in the server logs
• D3: the gateway routes traffic to servers

Our query is gateway five oh four. Gateway appears in two of three documents, so IDF is the log of one plus one point five over two point five, which is the log of one point six, about point four seven. The token five oh four appears in only one document, so IDF is the log of one plus two point five over one point five, about point nine eight.

• Query: gateway 504
• IDF(gateway) = 0.470
• IDF(504) = 0.981

Document one has length five. Its normalization factor is one minus point seven five, plus point seven five times five over six, which is point eight seven five, so K is one point oh five. Gateway appears twice: two times two point two over three point oh five is one point four four three, times point four seven gives point six seven eight. Five oh four appears once: two point two over two point oh five is one point oh seven three, times point nine eight one gives one point oh five three. The total is one point seven three one.

• K = 1.2 × 0.875 = 1.05
• gateway: 1.443 × 0.470 = 0.678
• 504: 1.073 × 0.981 = 1.053

Document two contains neither query term, so its score is zero, even though it says error three times. Document three has length six, so K is one point two. Gateway appears once: two point two over two point two is exactly one, times point four seven gives point four seven. The final ranking is D1, then D3, then D2.

• D2: 0 (no query terms)
• D3: 1.0 × 0.470 = 0.470
• Ranking: D1, D3, D2

Notice two effects. In document one, gateway went from one occurrence to two, but its saturated contribution rose only from about one point oh seven to one point four four, not double. And because document one is shorter than average, its K is smaller than document three's, which boosts every match. That is saturation and length normalization working exactly as designed.

• tf 1 to 2: 1.073 to 1.443
• Not double: saturation
• Shorter doc gets smaller K

BM25 fails on vocabulary mismatch. A user asks how to cancel my subscription, and the help page says terminate your plan. No shared words, no score. It also struggles with typos, word forms if you skip stemming, and questions phrased very differently from the source text.

• Vocabulary mismatch
• Synonyms and paraphrase
• Typos and morphology

Dense retrieval has the mirror-image problems. Rare identifiers, version numbers, and error codes are poorly represented, because a tokenizer chops them into pieces and the embedding blurs near-identical strings together. Specific names and domain jargon unseen in training can also land in the wrong neighborhood. Embeddings compress, and compression loses exact detail.

• Rare IDs and codes
• Exact names and numbers
• Out-of-domain jargon

The key insight is that the errors are different. When two systems fail on different queries, combining them can recover what each misses alone. This is the same logic as ensembling models. The remaining question is how to merge two ranked lists that come from totally different scoring systems.

• Different mistakes = good ensemble
• Union of candidates beats either alone

Why not just add the scores? Because they live on different scales. BM25 scores are unbounded and depend on the query and the corpus; a long query might give twenty-five. Cosine similarities usually sit in a narrow band, say point two to point nine. Add them raw and BM25 silently dominates, so your hybrid is just BM25 in disguise.

• BM25: unbounded, e.g. 0 to 25
• Cosine: roughly 0.2 to 0.9
• Cannot just add them

The elegant escape is to throw scores away and use only ranks. Rank one is rank one whether the score was twenty-five or point nine. Ranks from different systems are directly comparable, there is nothing to calibrate, and no assumptions about score distributions.

• Ranks are scale-free
• Only the order matters

Reciprocal rank fusion scores each document as the sum over all rankers of one over k plus the document's rank in that ranker. If a document is missing from a list, it contributes nothing. The constant k, conventionally sixty, controls how steeply the top positions are favored. Sort by the total, and you are done.

• RRF(d) = Σ 1 / (k + rank_i(d))
• k = 60 is conventional
• Missing from a list: contributes 0

Let us merge. The dense ranking is A, B, C, D. The BM25 ranking is C, A, E, B. With k equal to sixty, document A is rank one and rank two: one over sixty-one plus one over sixty-two, about point oh three two five two. C is rank three and rank one: one over sixty-three plus one over sixty-one, point oh three two two six.

• Dense: A, B, C, D
• BM25: C, A, E, B

Document B is rank two and rank four: one over sixty-two plus one over sixty-four, point oh three one seven five. E appears only in BM25 at rank three, giving point oh one five eight seven. D appears only in dense at rank four, giving point oh one five six three. Final order: A, C, B, E, D. Documents that both retrievers like float to the top.

• A 0.03252, C 0.03226
• B 0.03175
• E 0.01587, D 0.01563

What does k do? With a small k, like one, rank one is worth half and rank two a third, so a single first place can beat consensus. With a large k, all ranks look similar and agreement across lists matters more. Sixty is a widely used default that works well without tuning. You can tune it, but verify on your own evaluation set.

• Small k: top ranks dominate
• Large k: flatter, more consensus
• 60 is a robust default

The alternative is score fusion. Normalize each list, for example min-max to the range zero to one, then combine with a weight alpha. BM25 scores twelve, eight, two become one, point six, zero. Cosines point eight two, point eight oh, point seven eight become one, point five, zero. With alpha point five, you can then tune alpha on labeled queries.

• Min-max normalize each list
• score = α·dense + (1−α)·BM25

Which should you use? Rank fusion is robust and needs no tuning, but it throws away score gaps: a runaway winner and a narrow winner look the same. Weighted fusion keeps that information, but normalization is fragile, since one outlier distorts min-max. My recommendation is to start with RRF, and move to weighted fusion only if your evaluation shows a gain.

• RRF: robust, zero tuning
• Weighted: uses score gaps, needs tuning
• Start with RRF

One more family to name. Learned sparse retrievers, such as SPLADE, use a transformer to produce a sparse vector over the vocabulary, reweighting terms and even adding related words. They sit between BM25 and dense models. We will not go deeper today, but know they exist if plain BM25 is not enough.

• SPLADE: learned term weights
• Sparse vectors with expansion
• Named only; not covered today

Now the code. With the rank underscore bm25 package, you tokenize every document, build a BM25Okapi object, then call get scores with the tokenized query. Here a small helper lowercases and splits on non-alphanumeric characters. The key rule: use the identical tokenizer for documents and queries.

• Tokenize, then BM25Okapi
• get_scores returns one per doc

For the dense side, load any sentence embedding model, encode all documents once with normalized outputs, and encode the query at search time. Because vectors are unit length, a simple matrix product gives cosine similarity. Then argsort descending to get the ranking.

• Encode docs once, normalize
• Dot product = cosine

Reciprocal rank fusion is about six lines. Make a dictionary of scores. For each ranking, enumerate its document IDs starting rank at one, and add one over k plus rank. Finally sort the dictionary by value, descending. No libraries, no normalization, nothing to tune.

• Loop over rankings
• Accumulate 1 / (k + rank)

Test one: the query is E R R underscore five oh four. BM25 places the exact page at rank one, since that token is rare and has a huge IDF. The dense model puts it at rank six, behind generic gateway articles. After fusion, the page lands at rank one, because the BM25 vote is strong, and the dense list still contains it.

• Query: ERR_504
• BM25 rank 1, dense rank 6
• Hybrid: rank 1

Test two: how do I cancel my subscription. The help page says terminate your plan. BM25 only finds it through the shared word my or subscription somewhere deep in the list, say rank eight. The dense model understands the intent and ranks it first. Hybrid still surfaces it near the top. One pipeline, both wins.

• Query: cancel my subscription
• BM25 rank 8, dense rank 1
• Hybrid: rank 1 or 2

The first classic mistake is fusing raw scores. Adding a BM25 score of twenty-five to a cosine of point eight means the dense model has essentially no vote. And BM25 scores change from query to query, so even a fixed weight is unstable. Use RRF, or normalize per query before weighting.

• BM25 25.0 + cosine 0.8
• BM25 dominates
• Fix: ranks or normalization

The second mistake is tokenization mismatch in BM25. If you index lowercase tokens but pass the query as typed, E R R never matches err. If you split on underscores for documents but not for queries, the code never matches. Use one function for both sides, and decide deliberately how to treat hyphens, dots, and underscores in identifiers.

• Docs lowercased, query not
• Different stemming or splitting
• Punctuation in IDs

In practice, retrieve a generous top fifty from each retriever, fuse with RRF, and pass the top twenty or so to the cross-encoder from last lesson. Many vector databases and search engines offer hybrid queries natively, so check before building your own. And do not assume hybrid always wins; measure on your own queries.

• Retrieve top 50 from each
• Fuse, then rerank top 20
• Measure on your own queries

Recap. Keyword search wins on exact terms, dense wins on meaning. BM25 combines IDF, saturating term frequency controlled by k1, and length normalization controlled by b. Raw scores don't mix, so fuse ranks with reciprocal rank fusion, using k equal to sixty, or carefully normalize and weight. Use matching tokenizers, and always measure.

• BM25: IDF, saturating TF, length norm
• Fuse ranks with RRF, k = 60
• Same tokenizer both sides

But how do you know hybrid search actually helped? That is the next lesson: evaluating RAG, measuring retrieval and generation quality, and the common ways these systems fail. Thanks for watching.

• How do we know retrieval works?
• Recall, MRR, faithfulness
• Subscribe for episode 20