A subtle design choice hides inside every RAG system: when you retrieve, should you feed the model the raw passage from your documents, or a summary of it? Most systems never ask — they default to raw chunks and move on. But summaries and raw chunks each have a job, and matching them to the question is a real lever on both quality and cost. Here's how to think about when each wins, and the patterns (parent-document retrieval, summary nodes) that combine them.
What each one is good at
Raw chunks are the original text, verbatim. Their strength is fidelity: the model sees exactly what the source says, with no information lost or distorted by a summarization step. For precise, factual, detail-sensitive questions — exact numbers, specific policy wording, a precise definition, anything where the exact language matters — raw chunks are essential. A summary might paraphrase away the precise detail the answer hinges on. And a summary can introduce errors (summarization is lossy and occasionally wrong), which for factual retrieval is a real risk.
Summaries are condensed representations of larger content. Their strengths are coverage and efficiency:
- They let a small amount of retrieved text represent a lot of source material — useful when the answer needs the gist of something large.
- They're ideal for broad, high-level questions ("what is this document about?", "summarize the key themes") that no single raw passage answers well.
- They pack more conceptual content per token, which matters for context efficiency.
The trade is fidelity for breadth: summaries cover more but preserve less exact detail.
Match to the question type
The rule of thumb:
- Detailed/factual/precise question → raw chunks. "What's the exact refund window?" needs the source's exact words. Don't summarize away the fact you're trying to retrieve.
- Broad/overview/thematic question → summaries. "What are the main risks in this contract?" is a whole-document question that a summary node answers far better than a random precise passage.
Many systems face both kinds of questions, which is why the best patterns combine raw and summarized content rather than choosing one globally.
Pattern 1: parent-document (small-to-big) retrieval
A powerful hybrid: retrieve on small, precise units but return larger context. You embed and search over small chunks (precise matching — you find the exact right spot), but when a small chunk matches, you return its parent (the fuller surrounding section) to the model, so the answer has context. This isn't quite summaries-vs-raw, but it's the same instinct — decouple what you match on from what you feed the model. You get precision in retrieval and completeness in the context.
Pattern 2: summary nodes for hierarchical navigation
Build summaries at multiple levels — section summaries, a document summary — and embed those too, so they're retrievable. Now:
- A broad question retrieves a summary node (the right granularity for an overview).
- A specific question retrieves a raw leaf passage (the right granularity for a fact).
Retrieval matches the granularity of the question to the granularity of the content. Summaries also serve as efficient navigation: a query can hit a summary to identify the relevant region, then drill into raw chunks there. This is especially valuable for long documents where no single raw passage captures the whole.
Pattern 3: summaries as a routing/index layer
A more advanced use: summaries as a coarse index. Retrieve over summaries first to find which documents or sections are relevant (cheap, high-level), then retrieve raw chunks within those (precise). The summary layer narrows the space; the raw layer supplies the detail. This scales retrieval over large corpora by doing a cheap coarse pass before an expensive precise one — conceptually similar to the two-stage retrieve-then-rerank pattern, applied to granularity.
The costs of summaries
Summaries aren't free, which is why raw chunks are the sensible default:
- Generation cost at ingestion. Producing summaries (usually via an LLM) adds a heavier, more expensive ingestion step, and it's recurring — when content changes, summaries must regenerate.
- A new error source. A bad summary misleads retrieval and can inject wrong information into answers. Raw chunks can't lie about the source; summaries can.
- Maintenance. Summaries add content to keep fresh and consistent with their source.
So summaries earn their place where they add real value (broad questions, long documents, hierarchical navigation), not everywhere.
Practical guidance
- Default to raw chunks. They're faithful, cheaper to produce, and right for the majority of precise, factual questions most knowledge bases handle.
- Add parent-document retrieval when precise matching leaves the model short on context — a low-cost, high-value pattern that needs no summarization.
- Add summary nodes specifically for long, structured documents where users ask both pinpoint and whole-document questions. This is where summaries clearly win.
- Use summaries as a coarse index at large scale, to narrow before you retrieve raw detail.
- Don't summarize by default. For short, self-contained content, raw chunks are simpler and better; summaries there are cost and error risk with little upside.
The takeaway
Raw chunks give fidelity; summaries give coverage and efficiency — and the right choice depends on the question. Precise factual questions want the source's exact words; broad thematic questions want a summary that no single passage provides. The best systems don't pick one globally: they retrieve on precise units and return fuller context (parent-document), build multi-level summary nodes so retrieval matches question granularity, and reserve summarization for where it genuinely helps — long documents and coarse indexing at scale. Default to raw, reach for summaries deliberately, and you get fidelity where it matters and coverage where it counts, without paying the summarization tax everywhere.