Vector RAG is the default, and for good reason: embed your documents, retrieve by similarity, answer. It handles the overwhelming majority of "what does the doc say about X?" questions. But there's a class of question it handles badly — questions about relationships. "How does a change to service A cascade to the teams that depend on it?" isn't really about the content of any one document; it's about the connections between many. Graph RAG is built for exactly those questions, and knowing where the line falls saves you from forcing a graph onto a problem vectors already solve.
How each one models knowledge
Vector RAG represents knowledge as points in embedding space. Similarity is proximity. When you query, you find the passages nearest your question. There's no explicit notion of how documents relate to each other — only how similar their content is. It's fantastic at "find me text about this topic" and blind to "show me what connects to this entity."
Graph RAG represents knowledge as a graph: entities are nodes, relationships are edges. "Service A" depends on "Service B." "Alice" owns "the billing runbook." "Contract X" supersedes "Contract Y." Retrieval means traversing these edges. When you query, you can walk the graph — find an entity, follow its relationships, gather the connected context. The relationships are first-class data, not something the model has to infer from co-occurring text.
The questions where graphs win
Graph RAG earns its complexity on questions that are fundamentally about connections:
- Multi-hop relationship queries. "Which teams are affected if we deprecate this API?" requires following dependency edges outward. Vector search would retrieve documents mentioning the API, but assembling the dependency chain is exactly what edges are for.
- Aggregation over a structure. "Summarize everything related to this customer across contracts, tickets, and emails" benefits from a graph that links all those artifacts to the customer node, rather than hoping similarity surfaces them all.
- Global/holistic questions. "What are the major themes across this entire document set?" is where graph approaches with community detection shine — they build structure over the whole corpus rather than retrieving isolated passages.
The common thread: the answer depends on how entities relate, and those relationships are explicit and worth modeling.
The cost of the graph
Graphs aren't a free upgrade. The reason vector RAG dominates is that the graph's power comes with real overhead:
- Construction is expensive. Someone — usually an LLM — has to extract entities and relationships from your documents to build the graph. This is a heavier, more error-prone ingestion step than "chunk and embed." Bad extraction produces a wrong graph, and a wrong graph gives confidently wrong traversals.
- Maintenance is harder. When content changes, you don't just re-embed a chunk — you have to update entities and edges, which can ripple. Keeping a graph fresh is more work than keeping an index fresh.
- It's overkill for content questions. For "what does our refund policy say?" a graph adds nothing over vector search and costs more to build and run.
The pragmatic answer: often, both
The framing of "graph vs. vector" is usually a false choice. Mature systems tend to combine them: vectors for content retrieval (semantic similarity over passages), a graph layer for relationship traversal (structured connections), and a router that sends each query to the right mechanism — or uses the graph to inform which passages to retrieve.
Practically, most teams should:
- Start with vector RAG. It's simpler, cheaper, and answers most questions well. Hybrid search plus reranking gets you a long way.
- Add graph structure only where you've identified relationship-heavy questions vectors handle poorly — dependency analysis, entity-centric aggregation, holistic corpus questions.
- Route rather than replace. Keep vectors as the workhorse; use the graph for the specific queries that need it.
| Vector RAG | Graph RAG | |
|---|---|---|
| Models | Content similarity | Explicit relationships |
| Great at | "What does it say about X?" | "How does X connect to Y?" |
| Ingestion cost | Chunk + embed | Entity/relationship extraction |
| Maintenance | Re-embed changed chunks | Update nodes + edges |
| Best default | Yes | Only for relationship-heavy queries |
The decision isn't ideological. If your hard questions are about what documents say, vectors are enough and a graph is expensive ceremony. If they're about how things connect, and those connections are worth modeling explicitly, a graph turns unanswerable traversals into answerable ones. Most products need a lot of the first and a little of the second — so build the vector layer first, and reach for the graph exactly where relationships are the answer.