Multi-Hop RAG: Why One Retrieval Pass Isn't Enough

Multi-hop RAG is retrieval-augmented generation built for questions no single document can answer. Instead of one retrieve-then-generate pass, it retrieves, reasons over what it found, then retrieves again, chaining evidence across two or more hops until it has enough to answer. Standard RAG treats retrieval as a single lookup in front of generation. Multi-hop RAG treats retrieval as a loop the model controls, and that loop is also where most production failures in this pattern start.

I have shipped RAG pipelines that handled single-fact lookups cleanly and fell over the moment a question needed two connected facts instead of one. "Who now runs the company that bought the supplier Acme used before 2020?" cannot be answered by embedding that sentence and hoping one search returns everything. The query has no vector representation for the supplier's name, because that name is the answer to the first half of the question. Multi-hop RAG exists to close exactly that gap.

Key takeaways

If you read nothing else, read these.

• Multi-hop RAG chains retrieval across steps instead of retrieving once. It retrieves, reasons over the result, then retrieves again until the chain of evidence is complete.
• It exists for a specific failure: compositional questions. Single-hop retrieval fails when the second fact you need depends on a first fact you have not found yet.
• IRCoT, the technique that interleaves reasoning with retrieval, improved retrieval by up to 21 points and downstream QA by up to 15 points over one-step retrieve-and-read, tested across four multi-hop benchmarks.
• The failure mode is quiet, not loud. A bad document at hop two poisons the reasoning at hop three, and the final answer still reads as confident and grounded.
• Techniques built to improve multi-hop retrieval can make chain failures worse, not better, per 2026 robustness research on entity-graph linking and iterative query reformulation.

What is multi-hop RAG?

Multi-hop RAG is retrieval-augmented generation that answers a question by chaining two or more retrieval steps, using what it learns at each step to decide what to retrieve next. A single-hop system embeds the question once, retrieves the top-matching chunks, and generates an answer from that one batch of context. A multi-hop system treats the question as a chain of dependent sub-questions and resolves them one hop at a time.

What matters is not how the model decides to retrieve. It is what kind of question forces the loop. Multi-hop RAG is defined by compositional questions, ones where the second piece of evidence cannot be found until the first piece is already in hand. That is narrower than agentic RAG, which hands a model a full toolkit and lets it decide what to search and how many times, for almost any question. It also differs from active retrieval-augmented generation, which retriggers retrieval on the model's confidence mid-answer, not on an explicit dependency between facts. A system can be multi-hop without being agentic, a fixed pipeline that always runs a set number of hops, and agentic without being multi-hop, a model repeatedly searching for one fact it cannot pin down.

Why single-hop retrieval fails on multi-step questions

Single-hop retrieval assumes the question, as written, contains enough signal to find every fact the answer needs. That holds for direct lookups. It breaks once the question is compositional, because answering it requires an intermediate fact the query never named, since that fact is the output of an earlier step.

Take the supplier example again. One embedding of that sentence searches for something like "Acme supplier before 2020, company that bought it, current CEO" all at once. The retriever cannot search for the supplier's name specifically, because the corpus never states that name beside "bought" or "CEO." It needs the name first, and it cannot have it until a first retrieval step surfaces it.

This differs from the failure I cover in my piece on chunking strategies for RAG, where a fact splits across a chunk boundary inside one document. Here the fact is missing from the query itself, because the query was written before anyone knew which sub-fact it would need next.

If you are not sure whether your pipeline has this failure or a simpler recall problem, that diagnosis is the first thing the ViitorCloud custom AI and RAG systems team checks before adding a loop that costs more per query than the fix it was meant to justify.

Core techniques: query decomposition, IRCoT, and graph-based retrieval

Three approaches cover most production multi-hop RAG systems, and they differ in when the sub-questions get generated and how rigid the loop is.

Query decomposition breaks the question into an explicit sequence of sub-questions before retrieval starts, then answers each in order, feeding each answer forward into the next. It is the most inspectable version: you can review the plan before spending a retrieval call on it.

IRCoT, short for interleaving retrieval with chain-of-thought, does not plan the whole chain up front. It generates one reasoning sentence, retrieves based on that sentence, generates the next reasoning sentence from what came back, and repeats. Reasoning and retrieval take turns instead of splitting into a plan-then-fetch pipeline. Across HotpotQA, 2WikiMultiHopQA, MuSiQue, and IIRC, this interleaving improved retrieval by up to 21 points and downstream QA by up to 15 points over one-step retrieve-and-read, and it hallucinated less by grounding each step in freshly retrieved evidence (Trivedi et al., 2022).

Graph-based retrieval, the approach behind GraphRAG, takes a third route. It builds an entity graph over the corpus ahead of time, so the connections a multi-hop question needs, this person co-founded that company, that company acquired this one, already exist as edges instead of being discovered fresh at query time. It trades expensive upfront graph construction for cheaper traversal at answer time.

Whichever technique you pick, every hop still runs through the retrieval decisions I cover in my RAG tutorial: chunking, embeddings, hybrid search, and reranking. They just run once per hop instead of once per query.

How the multi-hop RAG retrieve-reason-retrieve loop actually runs

Strip away the differences between techniques and the architecture looks the same: select a sub-query, retrieve, reason over what came back, decide whether the chain is complete, and if not, generate the next sub-query from what you now know. Two design choices decide whether it holds up in production. Sub-question generation is either explicit, a planning step up front, or implicit, each reasoning step produces the next query as a byproduct, the way IRCoT does. The stopping condition decides when the chain has enough evidence, and getting it wrong either way is costly: stop early and you answer on an incomplete chain, keep going past sufficiency and you pay for hops that add nothing.

Here is an illustrative trace, shaped like the ones I instrument in production, not a real client log.

The value of logging it this way is not the trace itself. A broken chain and a correct chain produce the same final answer format: confident, cited, complete. The trace is the only place you see that hop three's confidence is meaningfully weaker than hop one's, and weak late-chain confidence is exactly where I have watched wrong answers slip through review undetected.

What the multi-hop RAG benchmarks actually show

Four benchmarks define how multi-hop RAG systems get evaluated, each built to punish a different shortcut instead of chaining evidence.

The point gains I cited earlier from IRCoT were measured against exactly this set, which matters more than the raw numbers: a benchmark built to resist shortcuts measures whether a system connected every document the question required, not whether it guessed the right entity from a surface pattern.

The failure mode: error compounding across hops

Every additional hop is another chance to retrieve a plausible but wrong document, and multi-hop systems do not fail loudly. They fail quietly.

Picture an illustrative case: hop two should find the acquirer of a specific supplier, and the corpus holds two similarly named companies acquired the same year. The retriever pulls the wrong one. Hop three reasons forward from that wrong company with full confidence, because nothing in the trace flagged the name. The final answer still cites sources and reads as grounded. It is just grounded in the wrong chain.

What makes this worse is that the fixes teams reach for first do not reliably help. A 2026 study testing entity-graph linking and iterative query reformulation, two common multi-hop extensions, across HotpotQA, 2WikiMultiHopQA, and MuSiQue, found both can raise raw retrieval accuracy while making the system more fragile to a single bad hop (a 2026 study on multi-hop retrieval robustness). Better retrieval and worse robustness came from the same change, because a confidently wrong path now has more evidence backing it, not less.

Name the trade-off directly: multi-hop RAG swaps single-point retrieval failure for compounding-chain failure. Your eval harness has to catch broken chains, not just grade the final answer, or you ship a system that scores better on your metrics while being wrong more expensively on the questions a customer cares about.

When multi-hop RAG is worth the cost, and when it isn't

Every hop is a retrieval call and, in most implementations, an extra model call to reason and generate the next sub-query. That cost lands on your inference bill every time the feature runs, not once in a demo.

Build multi-hop RAG when your queries are genuinely compositional, a wrong answer costs you a customer, and you have measured that single-hop retrieval fails on the patterns your users send. Skip it when most traffic is single-fact lookups, latency is tight, or you have not yet proven single-hop retrieval is what's broken.

One answer to "doesn't looping retrieval blow up cost" is EfficientRAG: a small, lightweight model generates the follow-up query and filters irrelevant retrieved content, instead of calling a large model at every hop, keeping per-hop cost far below a full LLM call (the EfficientRAG paper). It is not free. It is cheaper than the naive version, which is the point.

This is a revenue decision as much as an architecture one. A pipeline that adds seconds and a few cents per query is a good trade if it turns a wrong answer that costs a churned account into a right one. It is a bad trade if you are adding hops to questions single-hop retrieval already answered correctly.

Implementation checklist for production

Instrument these before you ship a multi-hop RAG system, not after the first wrong answer reaches a customer:

• Log every hop, not just the final answer: sub-query, retrieved documents, confidence score, and the reasoning step that consumed them.
• Cap the maximum number of hops. An unbounded loop is an unbounded bill and an unbounded chance to compound an error.
• Make the stopping condition explicit and testable. "Enough evidence" needs a definition you can eval against, not a self-report from the model.
• Evaluate the chain, not just the answer. A golden set that only grades final output misses a right answer reached through a broken chain, and misses a wrong answer reached through a chain that looked fine at every step but one.
• Track cost and latency per hop against a single-hop baseline. If the multi-hop version cannot earn its extra cost on your actual query mix, it has not earned a place in production.

What is multi-hop RAG and how is it different from regular RAG?

Multi-hop RAG retrieves, reasons over the result, and retrieves again, chaining two or more steps until it has enough evidence to answer. Regular RAG retrieves once, before generation starts, based only on the original question. The difference is whether retrieval can use what an earlier step found to decide what to look for next.

How does multi-hop RAG actually retrieve and reason across multiple documents?

It runs a loop: select a sub-query, retrieve, reason over what came back, and either generate the next sub-query from that new information or stop because the chain is complete. Techniques differ mainly in whether the sub-questions are planned up front, query decomposition, or generated one at a time as reasoning proceeds, the way IRCoT does.

Which benchmarks are used to evaluate multi-hop RAG systems?

HotpotQA, MuSiQue, 2WikiMultiHopQA, and IIRC are the four most common, and each penalizes a different shortcut: guessing from surface patterns instead of chaining evidence across the required hops.

Does multi-hop RAG cost more and run slower than standard RAG?

Yes. Every hop adds a retrieval call and typically a model call to reason and generate the next query. Lightweight approaches like EfficientRAG cut that cost with a small per-hop model instead of a large one, but multi-hop retrieval is never free. It pays for itself only on compositional questions single-hop retrieval was already getting wrong.

If you want the full retrieval discipline this sits on top of, chunking, hybrid search, reranking, and now multi-step chaining, my book Retrieval That Survives Contact walks through it end to end, including where each layer stops being worth the cost.