Hybrid Search vs Vector Search: When BM25 Still Wins
Hybrid search beats pure vector search in most production RAG, because BM25 catches the exact terms embeddings quietly misrank.
Hybrid search vs vector search is not a coin flip. For most production RAG systems, hybrid search wins, because it fuses BM25's exact-term matching with an embedding model's semantic matching. Pure vector search only earns its simplicity when your queries are consistently conversational and your corpus has no meaningful exact-match vocabulary: no IDs, no SKUs, no jargon a customer might type verbatim.
Picture a support-search tool built entirely on embeddings. It nails the paraphrased question, "why did my order get held," pulling the right refund-policy chunk even though the customer never typed the word "refund." Then a user pastes an actual error code, SKU-88214-A, and the same system returns three vaguely related paragraphs and nothing that contains that string. The embedding model was never trained to treat an alphanumeric SKU as anything but a rare, mostly meaningless token. This is illustrative, not a specific client outcome, but it is the exact shape of failure I see whenever someone ships vector-only search against a corpus that has any technical vocabulary in it.
Key takeaways
If you read nothing else, read these.
- Hybrid search beats pure vector search on most production corpora because it recovers the exact-term matches, IDs, SKUs, names, that embedding models silently misrank.
- On a financial text-and-table benchmark, BM25 alone (Recall@5 = 0.644) beat a frontier embedding model, text-embedding-3-large (Recall@5 = 0.587), and hybrid fusion via Reciprocal Rank Fusion beat both at 0.695.
- Reciprocal Rank Fusion combines results by rank position, not raw score, which sidesteps the scale mismatch between BM25 scores and cosine similarity.
- The fusion weight has no universal default. Pinecone recommends alpha = 0.75 for natural-language queries and alpha = 0.25 for keyword-heavy ones; Weaviate changed its own factory default in v1.24 because the old one left recall on the table.
- Hybrid search doubles your tuning surface, it does not remove tuning. You now own a sparse index, a dense index, and a fusion parameter, each of which needs evaluation against your own query mix.
What Hybrid Search Actually Fuses: BM25 and Embeddings
BM25 is a lexical ranking function. It scores a document by how often a query term appears in it, adjusted for how rare that term is across the whole corpus and how long the document runs. It has no concept of meaning: "cancel my subscription" and "terminate my plan" share zero terms, so BM25 treats them as unrelated. What it does exceptionally well is find the document containing the exact string you typed, whatever that string is.
Dense embeddings work the opposite way. A model converts your query into a vector and finds the documents whose vectors sit closest to it in a learned semantic space. "Cancel my subscription" and "terminate my plan" land near each other because the model learned they mean the same thing, with no words in common. What embeddings do not do well is anything the model never learned to place precisely: a product code, a rare proper noun, an internal ticket ID.
Hybrid search runs both retrieval methods against the same query, then merges the two ranked lists into one before anything reaches the model. Neither method gets disabled or demoted to a fallback. Both search the corpus every time, and the fusion step decides how much each contributes to the final ranking.
Where Pure Vector Search Quietly Fails
Vector-only retrieval fails hardest on queries carrying real lexical weight: SKUs, ticker symbols, case numbers, model names, exact-phrase legal or medical terms. An embedding model has seen millions of ordinary sentences during training and almost none of your specific product catalog. It has no reason to place "SKU-88214-A" anywhere useful relative to "SKU-88214-B," because both look, to the model, like arbitrary noise.
The failure is quiet because it does not look like an error. The system returns results, confidently, and they are wrong in a way that is easy to miss in a demo built on friendly, conversational test queries. It shows up the day a real user pastes an order number, a compliance officer searches a regulation section number, or a support agent looks up an error code straight from a log line.
If your corpus carries any of that vocabulary, product codes, ticket IDs, internal jargon, testing for this failure mode before launch is the cheapest insurance you will buy. That is exactly the gap a proper retrieval build audits for; see how ViitorCloud's RAG and knowledge-integration team evaluates hybrid indexing against your real query mix before it ships.
Where Pure BM25 Quietly Fails
BM25 fails in the mirror direction. It cannot bridge a vocabulary gap. A user who asks "how do I stop getting billed" gets nothing useful from an index built around the word "cancellation," because BM25 only rewards term overlap, and there is none here. Every synonym, every paraphrase, every conversational rewording of the same underlying question is invisible to a purely lexical index.
This is the failure mode most teams already know, because it is the one that pushed everyone toward embeddings in the first place. But moving all the way to vector-only search does not fix the SKU problem. It trades one blind spot for the other. Hybrid search is the answer to "why should I have to choose."
Reciprocal Rank Fusion: How the Combination Actually Works
The hardest engineering problem in hybrid search is not running two retrieval methods. It is combining their outputs into one ranked list, because BM25 scores and cosine-similarity scores live on incompatible scales. A BM25 score of 12 and a cosine similarity of 0.82 cannot be averaged; they are not the same kind of number.
Reciprocal Rank Fusion sidesteps that mismatch by discarding the scores entirely and fusing on rank position instead. Weaviate's implementation, described in its deep dive on fusion algorithms, scores each document as 1/(rank + 60): the top-ranked document from BM25 gets 1/61, the second gets 1/62, and so on, then the same math runs for the dense results, and the two scores are summed per document.
The constant, 60 in Weaviate's implementation, dampens the effect of rank differences further down the list, so a document ranked first versus fifth matters more than one ranked fiftieth versus fifty-fifth. A document that ranks moderately well in both lists usually beats a document that ranks first in one list and does not appear in the other at all. That consensus effect, across two independent retrieval signals, is the real mechanism behind hybrid search's gains.
Tuning the Alpha Weight for Your Corpus
Rank-based fusion is not the only option, and even within it, how much weight each method gets is a decision, not a constant. Pinecone's hybrid search documentation uses a convex combination instead of rank fusion: combined = alpha times dense plus (1 minus alpha) times sparse, where alpha ranges from 0 to 1. Alpha of 1.0 is pure semantic search; alpha of 0.0 is pure BM25-style lexical search.
Pinecone's own starting recommendation is alpha = 0.75 for natural-language, conversational queries, and alpha = 0.25 for queries with high keyword specificity: product SKUs, technical IDs, named entities. There is no single safe default across both query types, which means shipping one alpha for your whole corpus is already a bet on which kind of query dominates your traffic.
Weaviate frames the same decision differently: a choice between fusion algorithms, not just a weight. Its rankedFusion algorithm is the Reciprocal Rank Fusion described above. Its relativeScoreFusion algorithm instead normalizes the original BM25 and dense scores to a common 0-to-1 range and combines those, keeping score-magnitude information that pure rank fusion throws away. Weaviate switched its own factory default from rankedFusion to relativeScoreFusion starting in v1.24, because keeping score magnitude, not just rank order, measurably improved recall for many workloads.
The Numbers: When Hybrid Search Beats Vector Search Alone
The clearest evidence against "just use embeddings" comes from a benchmark most teams have not seen. On T2-RAGBench, a benchmark built from 23,088 queries against 7,318 financial text-and-table documents, BM25 alone posted a Recall@5 of 0.644, according to the paper "From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents." Text-embedding-3-large, a frontier dense embedding model, scored 0.587 on the same metric: worse than plain BM25. Hybrid fusion via Reciprocal Rank Fusion reached 0.695, roughly five points above BM25 alone and more than ten points above the embedding model by itself.
That result runs against the assumption most teams build on by default: that a newer, bigger embedding model is always the safer retrieval choice. Financial documents are full of the exact vocabulary embeddings misrank, ticker symbols, account numbers, line-item labels, table headers, so a corpus like that sits close to a worst case for vector-only search. I have watched the same pattern show up in a compliance search tool for a financial services team (illustrative, not a specific client outcome): analysts needed exact regulation section numbers and internal case IDs, not paraphrase matching, and a hybrid index measurably cut how often they fell back to manual keyword search in the original documents. The lesson generalizes past finance: any corpus with a meaningful density of exact-match tokens is a candidate for the same result.
The Failure Mode Nobody Mentions: Two Systems to Tune, Not One
Hybrid search does not remove tuning. It doubles it. You are now responsible for a sparse index, a dense index, and a fusion layer with its own parameter that has no universal correct value. Ship alpha = 0.5, or whatever the platform default happens to be, and walk away, and you will often underperform a single method someone actually tuned.
Weaviate changing its own factory default from rankedFusion to relativeScoreFusion is the proof: the team that builds the fusion algorithm still found its own old default was leaving recall on the table. If the vendor's default was wrong for long enough that they shipped a new one, your default is not safe just because it is the default.
The honest fix is the same discipline that governs every other retrieval decision: build a labeled eval set from your real queries, measure recall at each candidate alpha or fusion algorithm, and pick the one that wins on your corpus. That is retrieval evaluation work, not a one-time config choice, the same discipline I lay out for the rest of the pipeline in my retrieval-augmented generation tutorial and in why chunking sets your recall ceiling before retrieval even runs.
Fusion tuning fixes what you retrieve within a single pass. It says nothing about when you retrieve, which is a separate architecture question I cover in my pillar piece on active retrieval-augmented generation.
Hybrid Search vs Vector Search: A Decision Framework
Skip the debate and check your corpus against these four situations.
| Your situation | Best retrieval choice | Why |
|---|---|---|
| Corpus has SKUs, IDs, ticker symbols, or exact jargon | Hybrid (BM25 + embeddings) | BM25 recovers the exact-match hits embeddings silently misrank |
| Queries are consistently conversational, paraphrased, with no fixed vocabulary | Vector search alone can be enough | Semantic similarity already covers most of the recall you need |
| Corpus is almost entirely structured codes, names, or IDs with little natural language | BM25-only is often enough | Embeddings add cost without adding matching power |
| You cannot afford to tune and evaluate two systems plus a fusion layer | Pick the single method that wins on your eval set; do not default to hybrid | An untuned hybrid setup can underperform a well-tuned single method |
Is hybrid search always better than vector search for RAG?
No. Hybrid search wins on most production corpora because it recovers exact-term matches embeddings miss, but if your queries are consistently conversational and your corpus has no meaningful exact-match vocabulary, no IDs, SKUs, or jargon, pure vector search is simpler to run and tune, and the hybrid gain shrinks toward zero.
What is reciprocal rank fusion and how does it combine BM25 and embedding scores?
Reciprocal Rank Fusion combines two ranked result lists by scoring each document 1/(rank plus a constant, commonly 60) in each list, then summing the two scores per document. It ignores the raw BM25 and cosine-similarity scores entirely, sidestepping the fact that those scores live on incompatible scales and cannot be averaged directly.
When should I use BM25 instead of an embedding model?
Use BM25 alone, or weight it heavily in a hybrid setup, whenever your queries carry exact-match vocabulary an embedding model never learned to place precisely: product SKUs, ticker symbols, case numbers, internal jargon, or exact-phrase legal and medical terms. The financial-document benchmark above is the clearest evidence: BM25 alone beat a frontier embedding model outright on that corpus.
Does adding hybrid search slow down retrieval latency?
Yes, some. You run two retrieval passes, sparse and dense, instead of one, then a fusion step on top, which adds real but usually modest latency at reasonable index sizes. Measure it against your own latency budget before you ship. If you are already adding a reranker on top of hybrid retrieval, that stage typically costs more latency than the fusion step does.
The full discipline behind this decision, chunking, freshness, hybrid indexing, and the recall reviews that keep an index honest as your corpus moves, is what I walk through in my book, Retrieval That Survives Contact. Read that if you want the recall targets and the review cadence, not just the mechanism.