Reranking with Cross-Encoders: The Cheap Trick That Fixes RAG Search

Your vector search found the right document. It just put it at rank seven. Your prompt only has room for the top three, so the language model never sees the answer and confidently makes something up. Nothing is wrong with your embeddings. What is missing is a second opinion, and that is today's lesson: reranking with cross-encoders.

• Retrieved, but buried
• The LLM reads the top few

In the previous lesson we improved retrieval by rewriting queries so the search engine receives a better question. Today we improve what happens after the search returns. We take a rough candidate list and sort it much more carefully, so the best passages rise to the top before they reach the model.

• Last time: query rewriting
• Today: second-stage ranking

By the end you will know why a cross-encoder is more accurate than a bi-encoder, why you cannot use it for everything, and how to do the cost math. We will reorder five candidates by hand, write runnable Python that measures recall and MRR before and after, and cover the mistakes that quietly ruin reranking pipelines.

• Bi-encoder vs cross-encoder
• Cost math and latency
• Code with recall and MRR

Think about hiring. You do not interview a million applicants. A quick screen cuts them to fifty, and then you interview those carefully. Retrieval works the same way. Stage one is fast and cheap and aims to not miss the good candidates. Stage two is slow and precise, and it decides the final order.

• Stage one: fast and broad
• Stage two: slow and precise

Why not use one great model for everything? Because the accurate models are expensive, and the cheap ones compress meaning into a single vector, which loses detail. Every system trades speed against accuracy. Two stages let you get the speed of the cheap model over the whole corpus, and the accuracy of the expensive model on a tiny shortlist.

• Fast models lose detail
• Precise models cost too much

Notice the different goals. Stage one is judged by recall at fifty: is the right document anywhere in the shortlist? Stage two is judged by how high it puts the right document, using metrics like mean reciprocal rank. A reranker can only reorder what it receives, so it can never rescue a document that stage one missed.

• Stage one maximizes recall
• Stage two orders the shortlist

Recall from the embeddings lessons how a bi-encoder works. The query goes through the encoder alone and becomes one vector. Each document went through the encoder earlier, alone, and became its own vector. Relevance is just a cosine similarity between the two. The query and document never see each other inside the network.

• Query and doc encoded separately
• Compare vectors by cosine

A cross-encoder does something different. It concatenates the query and the document into a single input, separated by a special token, and runs the whole thing through one transformer. Every query token can attend to every document token, and the reverse, in every layer. The output is not a vector but a single relevance score.

• Query and doc concatenated
• Full attention across both

Why is that more accurate? A single vector must summarize a document for every possible question. A cross-encoder already knows the question, so attention can focus on the exact words that matter. Take the query, can I get a refund after thirty days. A bi-encoder sees a refund document. A cross-encoder can check whether the document actually mentions thirty days.

• Sees word-level interactions
• Handles negation and detail

Here is the catch. Because the score depends on the query and the document together, you cannot compute anything about the document in advance. There is no document vector to store in an index. For every new query, you must run a full forward pass for every pair you want to score. That one fact shapes everything else in this lesson.

• Score depends on the pair
• Doc vectors cannot be cached

Let us do the math, with an assumed number. Suppose a small cross-encoder takes about five milliseconds per query and document pair on a GPU with batching. That is an illustrative figure, so measure your own. Scoring one million documents means one million passes, which is five thousand seconds. That is over eighty minutes for a single query.

• 1,000,000 forward passes
• Assume 5 ms per pair

Now rerank only fifty candidates with the same model. Fifty times five milliseconds is two hundred fifty milliseconds, and batching can often cut that further. We went from eighty-three minutes to a quarter of a second, a factor of twenty thousand, just by letting the cheap stage shrink the problem first.

• 50 pairs x 5 ms = 250 ms
• Same model, same query

The first stage is cheap because the work is done ahead of time. At query time you embed the query once, which is one forward pass, and then an approximate nearest neighbour index like HNSW finds close vectors in a few milliseconds, as we saw in the index lessons. The expensive document encoding happened offline, one time, at ingestion.

• Embed query once
• ANN search in milliseconds

The reranking cost is roughly linear in the number of candidates, and it also grows with text length, since attention cost rises with the number of tokens in each pair. So the number of candidates is a dial. Turn it up for better recall into the reranker, turn it down for lower latency. Finding the sweet spot is an engineering decision you will make with measurements.

• Latency is linear in k
• Longer docs cost more

Training is simpler than you might expect. The simplest version uses binary relevance. You collect pairs labeled relevant or not relevant, pass each pair through the model, turn the score into a probability with a sigmoid, and minimize binary cross-entropy, exactly the loss from the ML course. Negatives are usually hard ones, documents that look relevant but are not.

• Binary relevance loss
• Labeled query-doc pairs

You can also train on order directly. A pairwise loss takes one relevant and one irrelevant document and pushes the relevant score above the other. A listwise loss looks at a whole list of candidates for a query and optimizes the ranking as a group. Many practical models are trained with simple binary labels, so do not feel you must pick a fancy loss.

• Pairwise: positive beats negative
• Listwise: whole ranking at once

Is there something between the two? Yes, late interaction, as in ColBERT. Documents are encoded alone, but into one vector per token instead of one per document, and those can be stored ahead of time. At query time, each query token finds its best matching document token, and those maximum similarities are summed. You get some of the cross-encoder's detail with precomputation, at the price of much more storage.

• One vector per token
• MaxSim late interaction

A newer option is to use a large language model as the reranker. You can ask it to score each passage, or show it the whole candidate list and ask for a ranking. These can be very strong, especially on hard or reasoning-heavy queries, and they follow instructions. But they cost far more per request, add noticeable latency, and can be inconsistent, so they should be used deliberately.

• Prompt an LLM to judge relevance
• Strong, but slow and costly

When is that worth it? Consider an LLM reranker when queries are complex, when the cost of a wrong answer is high, or when traffic is low enough that the extra seconds and money do not matter. For high-volume, latency-sensitive search, a small cross-encoder is usually the better default. As always, test both on your own data before deciding.

• Hard, nuanced queries
• Low traffic, high value answers

Time for a worked example. The query is, how do I reset my password. The bi-encoder returns five candidates by cosine similarity. First, password policy requirements, point seven four. Second, account security overview, point seven two. Third, steps to reset your password, point seven zero. Fourth, recovering a forgotten username, point six eight. Fifth, change your password in settings, point six six.

• Query: how do I reset my password
• Ranked by cosine similarity

Look at those scores. They are all within eight hundredths of each other, so the order is fragile. The document that truly answers the question, the reset steps, sits at rank three. Reciprocal rank is one divided by the rank, so for this query the score is one third. If your prompt kept only the top two, the answer would be lost.

• Right answer sits at rank 3
• Only slightly ahead of the rest

Now we run each pair through a cross-encoder. The raw outputs, called logits, are these. Password policy, minus zero point five. Account overview, minus two point one. Reset steps, eight point four. Forgotten username, minus three. Change in settings, four point two. Pass them through a sigmoid and you get roughly zero point three eight, zero point one one, nearly one, zero point zero five, and zero point nine eight.

• Raw scores are logits
• Sigmoid turns them to probabilities

Sort by the new scores. Reset steps goes to rank one, change your password in settings moves to rank two, policy requirements drops to three, account overview to four, and the username page to five. Reciprocal rank for this query jumps from one third to one. Notice the model understood that policy and security overview are about passwords, but do not answer how to reset one.

• Reset steps jumps to rank 1
• Settings doc rises to rank 2

One warning about those numbers. Cross-encoder scores are meant for ordering candidates for a single query. A score of four does not mean the same thing across different models, and you should not assume a fixed threshold works everywhere. If you want a cutoff, such as dropping weak passages, pick it empirically on labeled examples from your own data.

• Compare scores within one query
• Not across models or queries

The standard recipe is simple. Retrieve about fifty candidates from the vector index, rerank them with the cross-encoder, and keep the top five for the prompt. The wide net protects recall. The careful sort protects precision. The tight prompt saves tokens, cost, and distraction, because fewer irrelevant passages means fewer chances for the model to wander.

• Wide net, careful sort, tight prompt
• Typical shape of production RAG

Now budget the time. Say embedding the query and searching takes around twenty milliseconds, and reranking fifty candidates takes a couple hundred. These are example numbers, not benchmarks. The language model then spends seconds generating. So reranking is a modest slice of the total, and since it often shrinks the prompt, it can even save generation time and cost.

• Embed + search: tens of ms
• Rerank: about 100 to 300 ms
• LLM generation: seconds

If the budget is tight, you have levers. Score all candidates in one batched call on a GPU. Choose a smaller distilled model, such as a six layer MiniLM style model, which trades a little accuracy for a lot of speed. Cap the maximum length. Reduce the candidate count. And cache results for repeated queries. Always measure latency at the ninety-fifth percentile, not only the average.

• Batch pairs on GPU
• Use small distilled models
• Cache frequent queries

Now the code, using the sentence-transformers library. We load a bi-encoder, for example all-MiniLM-L6-v2, and a cross-encoder, for example cross-encoder slash ms-marco-MiniLM-L-6-v2. Then we build a tiny labeled set inside the script: a list of about twenty documents, and a few queries, each mapped to the index of its one relevant document.

• SentenceTransformer bi-encoder
• CrossEncoder reranker

Stage one reuses what we built before. Encode all documents once with normalize embeddings turned on, so a dot product equals cosine similarity. For each query, encode it, compute scores against the matrix, and take the top k indices with argsort. We keep k as a parameter, because we will compare retrieving ten against retrieving fifty.

• Encode docs once, normalized
• Top-k by dot product

Stage two is only a few lines. Build a list of pairs, each containing the query and one candidate's text. Call the predict method on the cross-encoder, which returns one score per pair. Then sort the candidate indices by those scores in descending order. Wrap it in a function that takes a query and a k and returns the reranked list of document indices.

• Build (query, doc) pairs
• ce.predict, then sort

Now measure. For each query, find the rank of its relevant document in a list. Recall at three is the fraction of queries where that rank is three or better. Mean reciprocal rank averages one divided by the rank. Compute both for the bi-encoder order, then for the reranked order of the top fifty, and print them side by side.

• Recall at 3: relevant in top 3?
• MRR: mean of 1 over rank

What should you see? Usually, mean reciprocal rank and recall at three improve after reranking, because the relevant document moves up. But recall at fifty cannot change, since reranking only reorders the same fifty. Your exact numbers depend on your data, and a toy set is only a sanity check. For decisions, build a larger labeled set from real user queries.

• Expect MRR and recall at 3 to rise
• Recall at 50 stays unchanged

Now the common mistakes. First, reranking too few candidates. If you retrieve ten and keep five, the reranker can barely change anything, and if the right passage was at rank twelve, it is gone. Retrieve widely, often thirty to a hundred, then cut. Check recall at your retrieval depth to see where the ceiling really is.

• Reranking top 5 fixes little
• Give the reranker room to work

Second, silent truncation. Cross-encoders have a maximum input length, commonly around five hundred twelve tokens for the query and document together. A long passage gets cut, and if the answer lives in the cut part, the score is wrong. Chunk sensibly, as in the chunking lesson, and check your model's limit and your chunk sizes.

• Models have a max sequence length
• The answer may be cut off

Third, ignoring latency. A cross-encoder that feels instant in a notebook can be painfully slow on a busy CPU server with long passages and many users. Measure at realistic load, track the slow tail, and set a timeout with a fallback to the first-stage order. A reranker that times out should degrade gracefully, not break the answer.

• Run on GPU or small model
• Test at realistic load

Let us recap. Reranking uses a fast first stage for recall and a slow second stage for precision. A bi-encoder embeds separately and can precompute, while a cross-encoder reads query and document together, scoring more accurately but costing a pass per pair. Retrieve fifty, rerank, keep five. ColBERT and LLM rerankers sit at other points on the tradeoff. Avoid too few candidates, truncation, and ignoring latency.

• Retrieve wide, rerank precisely
• Cross-encoders: accurate, not precomputable
• Measure MRR and recall

Next time we improve the first stage itself. Dense vectors are great at meaning but can miss exact keywords like error codes and product names, while classic BM25 is the opposite. In the next lesson, hybrid search, we combine both with reciprocal rank fusion.

• BM25 plus dense retrieval
• Reciprocal rank fusion