Embeddings are powerful, but they can miss an exact error code that old-school BM25 finds instantly.
Hook
Embeddings Failed
Your fancy embedding search just failed on a simple error code.
- Query: error E4012
- Wrong results returned
Problem
The Right Document
Say a user searches for error E4012. The right document contains that exact code.
- Exact code in the text
- Query matches literally
Dense
Meaning Space Blurs Codes
Embeddings turn text into meaning. But a code like that has almost no meaning to capture.
- Embeddings capture semantics
- Rare tokens get blurred
BM25
BM25 Counts Exact Matches
Keyword search, called BM25, does something simpler. It counts exact word matches.
- Rare terms score high
- Exact match wins
Rare Terms Win
A rare token like E4012 appears in few documents, so it gets a big weight. The right page jumps to rank one.
- Few documents contain it
- High weight, rank one
Tradeoff
Different Blind Spots
But keywords fail on paraphrases. Embeddings win there. Each method has blind spots.
- Keywords miss paraphrases
- Embeddings miss exact codes
RRF
Reciprocal Rank Fusion
So merge both rankings. Reciprocal rank fusion scores each document as one divided by k plus its rank, summed across lists.
- Score = sum of 1/(k + rank)
- Rewards agreement across lists
Outro
Next: Hybrid Search
Next lesson, we build full hybrid search. Follow so you don't miss it.
- BM25 plus dense
- Fused with RRF