What each approach is actually good at
Traditional RAG indexes chunks of text in a vector store and retrieves the top-k by semantic similarity. It shines on lookup questions — 'what's the refund policy?', 'which config flag controls TLS?' — where a single passage contains the answer.
GraphRAG extracts entities and relationships into a graph, then retrieves subgraphs plus supporting text at query time. It shines on questions that require joining facts from many documents — 'which customers on plan X have opened a ticket about integration Y in the last quarter, and what did we tell them?'.
The question-shape test
Before you build anything, write down 30 questions your users actually ask, in their own words. Then classify each one:
- Lookup: the answer lives in one passage. Vector RAG is enough.
- Multi-hop: the answer requires 2–4 documents that reference each other. Graph adds real accuracy here.
- Aggregation: the answer is a count, list or summary across many documents. Graph or a query engine — vector RAG will lie.
- Comparison: the answer contrasts two entities. Graph is a strong fit; vector RAG needs heavy prompting to compete.
Where GraphRAG budget really goes
Teams underestimate the extraction pipeline. Getting from 'a folder of PDFs' to 'a clean graph an LLM can query' takes an entity/relation schema, an extraction prompt, deduplication and merging, and a human-in-the-loop review for the first N thousand entities. Expect this to be 60% of your delivery time. It's also where you'll get most of your accuracy improvements — not from a better retriever.
The pragmatic 2026 architecture
The stack we deploy most often looks like this: one store (Neo4j) that holds both the property graph and the vector index. A batch extraction pipeline that runs on new documents, produces triples with provenance back to the source chunk, and emits a diff for review. A retriever that does a vector search first, hops through the graph for two levels, and returns a subgraph plus the source chunks. The LLM answers over that combined context.
// Retrieve subgraph + supporting chunks for a query embedding
CALL db.index.vector.queryNodes('chunk_embeddings', 8, $query_embedding)
YIELD node AS seed_chunk, score
MATCH (seed_chunk)-[:MENTIONS]->(e:Entity)
OPTIONAL MATCH (e)-[r*1..2]-(neighbor:Entity)
OPTIONAL MATCH (neighbor)<-[:MENTIONS]-(supporting:Chunk)
RETURN
collect(DISTINCT seed_chunk) AS seeds,
collect(DISTINCT e) AS entities,
collect(DISTINCT neighbor) AS neighbors,
collect(DISTINCT r) AS relationships,
collect(DISTINCT supporting)[..12] AS supporting_chunks;Evaluation: the only thing that matters
You can't compare RAG approaches on vibes. Build a labelled evaluation set of 100–300 real questions with expected answers and expected sources. Include multi-hop and aggregation questions — this is where the two approaches diverge, and where a naive benchmark will lie to you.
Score on three axes: answer correctness (LLM-graded against the reference), citation correctness (did the answer cite the right sources?) and latency at p95. Report all three every week during the build. The graph almost always loses on latency and wins on the other two; the trade-off has to be made explicitly.
When to stay on vector RAG
If your questions are 90% lookup, if your corpus is under a few thousand documents, or if you don't have the eval investment to prove the graph is worth it — stay on vector RAG. Spend the same time on chunking strategy, query rewriting and reranking. You'll get 80% of the graph's benefit for 20% of the cost.
FAQ
- How much data do I need before GraphRAG pays off?
- Rule of thumb: a few thousand documents that reference shared entities. Below that, vector RAG plus a good reranker is faster to ship and hard to beat.
- Can I use OpenAI / Anthropic for extraction, or do I need a fine-tuned model?
- Frontier models with a structured-output schema are the pragmatic default in 2026. Fine-tuning only pays back if your entity types are unusual or your corpus is very large.
- What breaks when we scale to millions of documents?
- Extraction cost and graph-write throughput, in that order. Both are solvable with batching and idempotent upserts — but plan for it on day one, not day one hundred.
Building something like this?
Simatech ships AI-native Scrum squads for teams doing exactly this work — MCP agents, GraphRAG, LLMOps and the rest.
Talk to us