What Reranking for RAG Actually Fixes (and Costs)

Reranking for RAG adds a cross-encoder scoring stage after initial retrieval. It jointly encodes the query and each candidate document, then reorders the top results by true relevance instead of trusting the vector search ranking as final. Done right, it lifts NDCG@10 and recall by real double-digit margins for well under 200ms of added latency per query. Done on top of a retriever that never surfaced the right chunk, it fixes nothing and just adds cost.

I have shipped this stage into production RAG pipelines more times than I can count, and the pattern repeats. A team ships hybrid search, sees the demo answer well, then watches accuracy plateau below what the eval set says should be possible. Reranking is usually the fix. It is also, just as often, the thing a team bolts on to paper over a retrieval problem reranking was never built to solve.

Key takeaways

• Reranking reorders, it doesn't retrieve. A cross-encoder scores the candidates your first-stage retriever already pulled, commonly the top 50 to 100. It cannot surface a chunk that never made that set.
• The accuracy gain is real and well documented. Adding a cross-encoder reranker on top of hybrid retrieval produced a 17.2 percentage-point gain in MRR@3 and a 12.1 point gain in Recall@5 over unreranked hybrid search in one text-and-table benchmark.
• The latency cost is small per query but compounds at scale. A cross-encoder pass typically adds tens to low hundreds of milliseconds; watch cost and latency multiplied by query volume, not the per-call figure alone.
• A reranker is a maintained dependency, not a one-time model pick. Cohere deprecated rerank-3.5 and auto-routed traffic to a successor model with different score calibration, which silently breaks any hard-coded relevance threshold.
• Not every query needs it. Queries the retriever already ranked correctly gain nothing from a reranking pass and just pay the extra latency, which is why routing reranking selectively is now an active research area.

What reranking actually does in a RAG pipeline

Reranking is the second stage of a two-stage retrieval pipeline. The first stage, usually a bi-encoder or hybrid keyword-plus-vector search, casts a wide net and returns a candidate set fast, often the top 50 to 100 chunks. The second stage, the reranker, re-scores each of those candidates against the query with a slower, more accurate model before handing the final top-k to the generator.

The split exists because you cannot afford the accurate model's cost across your whole corpus. A cross-encoder that jointly reads a query and a document scores well, but running it against every chunk in a million-document index would be too slow to serve a request. So the pipeline uses a cheap, approximate method to shrink the field, then spends the expensive, accurate method only on candidates that survived the cut.

Why bi-encoder vector search puts the right chunk in the top 50, not rank 1

A bi-encoder, the model behind most vector search, embeds the query and every document independently, then compares the resulting vectors with cosine similarity or dot product. That independence is what makes it fast: you precompute document embeddings once and compare against a new query embedding with simple vector math, indexed for sub-second lookup across millions of vectors.

It is also what caps its accuracy. The query and document embeddings are never seen together by the model, so two chunks that use different words for the same idea can end up far apart in vector space, while two chunks that share surface vocabulary but differ in meaning can end up close together. A bi-encoder is good at getting the right answer somewhere into a shortlist. It is measurably worse at putting that answer in first place, because ranking within a shortlist requires comparing candidates against each other and the query jointly, which a bi-encoder cannot do by construction. That gap, between "in the candidate set" and "at rank one," is the entire reason reranking exists as a separate stage rather than something a better embedding model fixes on its own.

If you are trying to work out whether your pipeline's accuracy ceiling is a ranking problem or a recall problem before spending engineering time on either, that diagnose-first approach is how ViitorCloud approaches RAG and retrieval builds: measure which stage is capping accuracy before fixing the wrong one.

How a cross-encoder scores query-document pairs

A cross-encoder takes the query and one candidate document, concatenates them, and passes the pair through a transformer together, so every attention layer compares query tokens against document tokens directly. The output is a single relevance score for that pair, not an embedding to compare later. That joint pass is why cross-encoders consistently outscore bi-encoders on ranking quality: they answer "how relevant is this document to this query," not "how similar are these two independently computed points."

The cost of that accuracy is throughput. A bi-encoder embeds each document once, ever. A cross-encoder runs a full forward pass for every query-document pair, at query time, because the score only exists for that pairing, so scoring 100 candidates means 100 forward passes, every request. That is why cross-encoders rerank a shortlist instead of scoring an entire index: the architecture that makes them accurate is what makes them too slow to run as your primary retriever.

ColBERT-style late interaction splits the difference: it encodes tokens independently, like a bi-encoder, but keeps token-level vectors and compares them at query time instead of collapsing to one vector per document. It recovers some cross-encoder precision at closer to bi-encoder speed, at the cost of a heavier, per-token index.

Choosing a reranker: hosted API vs. self-hosted cross-encoder

Three options cover almost every production case, and the choice depends more on data residency and query volume than on a small accuracy delta between them.

I default clients toward a hosted reranker for a pipeline's first production version, because the accuracy-per-engineering-hour is hard to beat. Once query volume or compliance requirements make the per-call cost or the data-residency question real, self-hosting BGE reranker v2-m3 is the standard fallback: same cross-encoder mechanism, no external call, no per-query bill.

What reranking costs: latency and dollars per query

Budget the reranking stage in two currencies: latency, since a cross-encoder pass adds real milliseconds on the critical path of every request, and cost, since a hosted API charges per search while a self-hosted model still consumes GPU time you pay for whether or not it is busy. At Cohere's published rate of $2.00 per 1,000 searches, the arithmetic is simple until your query volume isn't. A pipeline serving a few thousand queries a day barely notices the bill; a pipeline serving a few million does, and that is the volume where self-hosting a model like BGE reranker v2-m3 usually pencils out cheaper, once you account for the GPU capacity you need on hand regardless.

Here is an illustrative before-and-after, shaped like the traces I instrument on client pipelines, not a real production log.

The bi-encoder already did its job here: the right chunk was in the top three. The reranker's job was smaller and more specific, moving it to rank one so the generator sees it first. That is a narrower win than "reranking finds the answer," and worth being precise about, because the narrower framing is what tells you when reranking will and won't help.

The failure mode: reranking can't fix a candidate set that never had the answer

A cross-encoder only reorders the candidates it is handed, typically the top 50 to 100 from first-stage retrieval. If the correct chunk fell outside that set, because of a bad chunk boundary, a stale index, or a first-stage retriever that missed it entirely, no amount of reranking surfaces it. The reranker has nothing to promote.

I have watched teams add a reranker before fixing chunking or first-stage recall, and the result is flat production accuracy at a higher latency and API bill. The reranker's own benchmark numbers looked great; the team's actual query numbers barely moved, because the bottleneck was recall into the candidate set, not ranking quality inside it, a problem reranking was never built to solve. Chunking usually sets that ceiling first, which is why I treat it as the earlier, more foundational fix; I cover it in my piece on chunking strategies for RAG.

The diagnostic is simple: check whether the correct chunk appears anywhere in your first-stage candidate set, at any rank, across your eval queries. If it is missing outright on a meaningful share of failures, reranking will not move your accuracy number; fix retrieval first. If it is present but buried below the top handful, reranking is exactly the right tool.

A minimal eval harness for whether reranking is actually helping

Do not ship a reranker on faith that cross-encoders are "more accurate" in general. Measure it against your own corpus and queries, with a harness this small:

• Recall@50 before reranking. Is the correct chunk in the candidate set at all? This tells you whether retrieval, not reranking, is your real bottleneck.
• NDCG@10 and MRR@3, before and after. The delta is the entire case for adding the stage; a small delta means the reranker isn't earning its latency.
• Latency added per query, measured at your real candidate-set size, not a toy benchmark's default.
• Cost per 1,000 queries at your actual and projected volume, hosted or self-hosted.
• A held-out slice the retriever already ranked correctly. Reranking should not make these worse; if it does, the reranker is miscalibrated for your domain, not just slow.

Run this on a golden eval set of 100 to 300 real queries with known correct chunks before deciding reranking earns its cost, and rerun it whenever you change the reranker model. What a reranker was trained to optimize matters as much as its architecture: the InfoGain-RAG line of work found that a reranker trained specifically on which documents raise answer accuracy, rather than on generic relevance labels, improved exact-match accuracy on NaturalQuestions by 17.9 percent over naive RAG and 12.5 percent over standard ranking-based RAG baselines.

When to skip reranking entirely

Skip it when your retriever already puts the right chunk at or near rank one on your eval set; reranking a candidate set that is already correctly ordered adds latency and cost for accuracy you already had. That is not a hypothetical: research on adaptive reranking has found that a one-size-fits-all reranking pass is computationally wasteful for queries the retriever already ranked correctly, which is why routing only the queries that need reranking is an active research direction rather than applying it to every request.

Skip it, too, when retrieval recall is the real problem. A reranker on top of a retriever that only surfaces the right chunk 40 percent of the time will make that 40 percent look slightly better and leave the other 60 percent untouched. Fix chunking and first-stage recall first.

Keep it when your eval harness shows the correct chunk landing outside the top few positions on a meaningful share of queries, and the candidate set otherwise contains it. That gap between "present" and "prioritized" is precisely what a cross-encoder closes, and it is where the accuracy numbers in this piece come from.

If you want the retrieval-specific version of this discipline end to end, from chunking through hybrid search through the reranking decision, my book Retrieval That Survives Contact walks through where each stage sets your accuracy ceiling and where the next one can and can't raise it. For how retrieval decisions move inside the generation loop itself, not just before it, see my piece on active retrieval augmented generation.

For the full pipeline this reranking stage slots into, the retrieval augmented generation tutorial walks the build from ingestion to answer.

If you are past the point of guessing and need a two-stage retrieval pipeline, reranker included, instrumented with the evals that prove it earns its latency, that measure-before-you-add-the-stage discipline is what ViitorCloud's custom AI solutions team builds into RAG and retrieval systems: recall diagnosed first, reranking added only where the numbers say it will pay for itself.

What is reranking in RAG and how is it different from retrieval?

Retrieval is the first stage: a fast, approximate search, usually vector or hybrid, that pulls a broad candidate set from your full corpus. Reranking is the second stage: a slower, more accurate cross-encoder that re-scores just those candidates by jointly reading the query and each document, then reorders them so the most relevant reach the generator first. Retrieval decides what's in the running. Reranking decides what goes first.

Do I still need a reranker if I'm already doing hybrid search?

Usually yes, if your eval harness shows the correct chunk landing in the candidate set but not near the top. Hybrid search improves recall into the candidate set; it does not add the joint query-document comparison a cross-encoder uses to rank within that set. The two stages solve different problems and commonly stack: hybrid search widens what gets considered, reranking sharpens the order of what was found.

How much latency does a cross-encoder reranker add to a RAG pipeline?

Commonly tens to a couple hundred milliseconds per query, depending on candidate-set size and model choice, well under 200ms in most production configurations. What matters more than the per-query figure is total latency and cost at your real query volume: a pass negligible at pilot scale can become a meaningful line item once you're serving millions of requests a month.

Should I use a hosted reranker like Cohere Rerank or self-host something like BGE reranker v2-m3?

Start hosted if you want strong accuracy without standing up infrastructure, and budget for the per-search cost and for the reality that hosted models get deprecated and migrated on the provider's schedule, not yours. Move to self-hosting BGE reranker v2-m3 or a similar Apache-2.0 cross-encoder once query volume makes the per-call cost material, or once you cannot send query data to a third-party API for compliance reasons. Both are genuine cross-encoder architectures; the choice is about ownership and economics, not accuracy alone.