Graph RAG: A Bet on Multi-Hop, Not a Universal Upgrade

Graph RAG replaces or augments vector similarity search with a knowledge graph: entities and relationships pulled from your corpus, clustered into communities, and traversed at query time instead of matched by embedding distance. It earns its cost on one specific query shape, questions that require connecting facts across many documents, not the single most similar chunk. On simple factual lookup, plain vector RAG still wins on speed and accuracy. This is not a universal upgrade to RAG. It is a bet you make for multi-hop and global sensemaking queries, and a bet you lose if you make it for anything else.

I have watched teams reach for graph RAG the moment a multi-document question breaks their pipeline, then discover the real cost sits upstream of the query, in building and maintaining the graph itself. Microsoft's original GraphRAG paper, the one that made this pattern a mainstream conversation, is explicit that the technique earns its keep on "global sensemaking" questions over large corpora. It says nothing about your support tickets or your product FAQ, where one retrieved chunk already answers the question.

Key takeaways

If you read nothing else, read these.

• Graph RAG trades chunk similarity for entity and relationship traversal. It builds a knowledge graph from your corpus, clusters it into communities with the Leiden algorithm, and answers queries by walking that structure instead of matching embeddings.
• It wins on multi-hop and global sensemaking queries. On the MultiHop-RAG benchmark, graph-guided retrieval scored 70.3 versus 67.0 overall accuracy against vector-based retrieval.
• Plain vector RAG still wins on single-hop factual lookup. A 2025 Michigan State and Meta study found plain RAG ahead of the best graph method on that query type, F1 64.8 versus 63.0.
• Accuracy is implementation-dependent, not a fixed win. Diffbot's original graph RAG benchmark measured 56.2% accuracy; FalkorDB's 2025 SDK pushed the same benchmark past 90%, while vector RAG scored 0% on schema-bound queries like KPIs and forecasts.
• Graph construction is the hidden tax. LLM-based entity extraction runs every time you add or re-index data, not once, and a meaningful share of extracted relationships turn out to be noise rather than signal.

What is Graph RAG?

Graph RAG is a retrieval architecture that builds a knowledge graph from your source documents, entities and the relationships between them, then clusters that graph into communities and pre-generates a summary for each one. At query time, the system traverses the graph or reads community summaries instead of matching an embedding to the nearest chunk.

Vector RAG stores every chunk as an embedding and retrieves by similarity. Graph RAG stores entities as nodes and relationships as edges, then retrieves by walking that structure or handing the model a pre-written summary of an entire community of related entities. The unit of retrieval changes from "a chunk that looks similar" to "a set of facts that are connected."

How Graph RAG differs from vector RAG

The two architectures answer a different question at index time and a different question at query time. A table makes the contrast concrete.

Neither column is free. Vector RAG's simplicity is also its ceiling: it cannot connect two facts that never appear in the same chunk. Graph RAG's structure is also its overhead. You pay for it at every index update, whether or not that day's queries needed it.

How to build a knowledge graph for RAG

Building the graph is a pipeline in itself, run before any user asks a question. Four steps, in order.

• Entity extraction. An LLM reads each chunk and pulls out named entities: people, organizations, products, dates, claims. This step costs money, because it runs an LLM call over your whole corpus, not once but every time the corpus changes.
• Relationship mapping. The same pass, or a second one, extracts relationships between entities, "acquired," "reports to," "supersedes," "caused." Each becomes an edge in the graph.
• Community detection. The Leiden algorithm clusters the graph into hierarchical communities: groups of entities more connected to each other than to the rest of the graph. Microsoft's original GraphRAG method uses this step to turn a flat entity graph into a navigable hierarchy.
• Community summarization. An LLM pre-writes a summary for each community before any query arrives. At query time the system reads these summaries instead of re-deriving them, which is what makes global sensemaking questions answerable in reasonable time.

Picture a compliance team with two thousand vendor contracts, amendments, and side letters. Someone asks which contracts inherit the liability clause a master agreement's Q2 addendum changed. No single chunk contains that answer. It requires connecting the master agreement, the addendum, and every contract that references it by name. This is an illustrative shape of query, not a specific client's, but it is the shape graph RAG's community structure is built to answer, and the shape a chunk-similarity search walks straight past.

If you are deciding whether that build is worth it before committing engineering weeks to it, that is the question ViitorCloud's custom AI and RAG systems team answers first: prove the query shape needs graph traversal before paying for the extraction pipeline that produces one.

When Graph RAG actually beats vector search

Two query shapes justify the cost. Multi-hop reasoning is a question that needs facts from more than one document connected in sequence, "which vendor supplies the part that failed in the recall the FDA cited last quarter." Global sensemaking is a question about the whole corpus, not one document in it, "what are the recurring themes across every incident report this year." Neither has a single chunk that answers it.

The numbers back a real, bounded advantage, not a blanket one. On the MultiHop-RAG benchmark, graph-guided retrieval scored 70.3 versus 67.0 overall accuracy against vector-based retrieval. A 2025 controlled study from Michigan State and Meta ran plain RAG against four GraphRAG families and found plain RAG ahead on single-hop factual lookup, F1 64.8 versus 63.0 for the best graph method. No single architecture won across every query type they tested.

That variance shows up sharply in FalkorDB's 2025 benchmark. Diffbot's original graph RAG method measured 56.2% accuracy on enterprise queries. FalkorDB's newer GraphRAG SDK pushed the same benchmark past 90%. Vector RAG scored zero percent on schema-bound queries in that test, KPIs and forecasts, where graph structure was the only architecture that recovered any signal at all. Graph RAG is not one technique with one accuracy number. It is a family of implementations, and the gap between the worst and best of them is larger than the gap between graph RAG and vector RAG at their averages.

The frameworks: Microsoft GraphRAG vs. LightRAG vs. Neo4j/FalkorDB

Microsoft's GraphRAG is the reference implementation most of this category traces back to. It builds the graph with an LLM, clusters it with the Leiden algorithm, and pre-generates community summaries at every level of that hierarchy. On corpora in the one-million-token range, the paper reports substantial improvements in the comprehensiveness and diversity of answers to global sensemaking questions, compared to a conventional RAG baseline. It is thorough and it is expensive to run, because every level of the hierarchy gets its own LLM-written summary.

LightRAG positions itself as the lighter alternative: it simplifies entity and relationship extraction and adds a dual-level retrieval mode instead of a full community hierarchy. In the project's own published benchmarks, LightRAG outperforms GraphRAG on comprehensiveness across several test domains, for instance 54.4% versus 45.6% on an agriculture-domain test set, while emphasizing faster, cheaper incremental updates as its main selling point over Microsoft's reference design.

Neo4j and FalkorDB sit a layer below both. They are graph databases, not retrieval frameworks: the place you store and query the graph once you outgrow a research reference implementation, with native traversal and a query language built for exactly this kind of connected lookup. Most production graph RAG systems pair one of these with a construction pipeline, rather than choosing a database instead of a framework.

The cost and latency nobody puts in the demo

Graph construction is the hidden tax. LLM-based entity extraction over a real corpus is slow, and it runs every time you add, re-index, or update data, not once at query time like an embedding pass. A meaningful share of extracted relationships turn out to be thematically irrelevant noise. Filtering out low-frequency triples has been shown to improve accuracy in some setups, which means the graph was carrying noise, not signal, before anyone filtered it.

At query time, graph traversal also adds latency over pure vector search, worse for aggregation and path-finding queries than for single-entity lookups. That cost is a revenue decision as much as an engineering one. I have killed a graph RAG build for exactly this reason: the support team's actual queries were almost entirely single-entity lookups, "what's our refund policy for this SKU," a query shape vector search already handled at a fraction of the latency and the indexing cost. The graph would have added weeks of build time and an ongoing extraction bill to answer questions nobody was asking.

Per the GraphRAG-Bench framing that ICLR 2026 submissions have converged on, the technique pays off "precisely when questions demand reasoning across pieces," not otherwise. That is a narrow, specific bet. Name it as one before you fund it as a general upgrade.

A decision framework: do you need Graph RAG at all?

Run your actual query logs against this checklist before you build anything.

• Most queries are single-fact lookups. If a customer asks "what's the price of X" or "when did Y ship," vector RAG already answers this correctly and faster. Don't build a graph for it.
• Some queries need facts connected across documents. If answering requires two or three hops through separate sources, a knowledge graph is the architecture that recovers what chunk similarity misses.
• A few queries ask about the whole corpus. "Summarize the recurring themes across this year's reports" has no single chunk that answers it. This is the query shape Microsoft's GraphRAG was built for.
• Your corpus changes faster than you can afford to re-extract it. If entities and relationships shift daily, the extraction cost recurs daily. Budget for that before you budget for the graph.
• You can measure the gain before you ship it. Run the same eval set through vector RAG and a graph RAG prototype. If the accuracy delta doesn't clear the added cost and latency, don't ship the graph.

Most of the discipline here is identical to what governs any RAG decision. It starts with how you chunk the source corpus, and it should sit inside the six decisions that make up a production RAG pipeline, not replace them. Graph RAG changes the storage and retrieval layer. It does not remove the need for the rest of the stack, or the evals that prove it works.

That is a different axis of change than the one I covered in active retrieval augmented generation, which moves the retrieval decision's timing during a single answer, not the data structure it retrieves from. You can, in principle, combine both: an active retrieval loop that decides when to fetch, reading from a graph instead of a flat vector index. Most teams should get one of the two working and measured before reaching for both.

What is Graph RAG and how does it work?

Graph RAG is a version of retrieval-augmented generation that stores your corpus as a knowledge graph, entities as nodes and relationships as edges, instead of as embedded text chunks. It works by extracting entities and relationships with an LLM, clustering them into communities, and answering queries by traversing that graph or reading pre-written community summaries, rather than matching embeddings to find the nearest chunk.

Is Graph RAG better than regular vector-based RAG?

Not universally. Graph RAG measurably beats vector RAG on multi-hop and global sensemaking queries, where facts from several documents need to be connected. Plain vector RAG measurably beats graph RAG on single-hop factual lookup, where one chunk already contains the full answer. Which one wins depends entirely on your actual query shape.

How much does it cost to build and maintain a knowledge graph for RAG?

More than most demos show. Entity and relationship extraction requires an LLM pass over your entire corpus, and that pass reruns every time you add or update data, not once at setup. Query-time traversal also adds latency over a single vector similarity search. Lighter frameworks like LightRAG target lower construction and update cost than Microsoft's reference implementation, but the cost is never zero, and it recurs with every re-index.

Do I need Graph RAG, or is vector search enough for my use case?

Check your actual queries first, not your ambitions for the product. If most questions are single-fact lookups, vector RAG is enough and cheaper to run. If a meaningful share of your queries require connecting facts across documents or summarizing an entire corpus, graph RAG earns its cost. Measure the accuracy delta on your own eval set before deciding either way.

If you want the full retrieval discipline this sits inside, freshness, chunking, hybrid search, and now the choice of storage structure, my book Retrieval That Survives Contact walks through it end to end. If you would rather have a team prove which architecture your query logs justify before committing to either one, that is exactly what ViitorCloud builds for custom AI and RAG systems: the eval harness that answers the question before the graph gets expensive.