Flat chunking — split a document into a list of equal pieces — works fine for short, self-contained content. It breaks on long documents. A 200-page contract, a textbook, a research paper: these have deep structure and long-range dependencies, and a flat list of chunks throws all of that away. You retrieve a precise passage but lose the context it depends on, or you make chunks big enough to hold context and lose retrieval precision. Hierarchical chunking resolves this tension by keeping the document's structure as the chunk structure — and summary nodes extend it to whole-document questions.
The precision-vs-context tension
Every chunking decision on long documents fights the same trade-off:
- Small chunks are precise — the embedding represents one tight idea, so retrieval is accurate. But a small chunk is stripped of context: it's a paragraph with no idea which section or chapter it belongs to, and long documents are full of passages whose meaning depends on that surrounding context.
- Large chunks carry context but dilute precision — the embedding averages several ideas, so retrieval gets fuzzy, and you burn context budget on material the query didn't need.
On a short doc this barely matters. On a 200-page document it's fatal: you cannot pick a single chunk size that is both precise enough to retrieve well and large enough to preserve the context those passages need. Hierarchical chunking sidesteps the choice.
Parent-child chunking
The core pattern: chunk at multiple granularities and link them.
- Child chunks are small and precise — you embed and retrieve on these, so retrieval stays accurate.
- Parent chunks are larger (the enclosing section or a bigger window) — after retrieving a child, you return the parent (or the child plus its parent context) to the model, so the answer arrives with the context it needs.
retrieve on: small child chunk (precise match)
return to LLM: parent section (child + surrounding context)
You get the best of both: the precision of small chunks for finding the right spot, and the context of large chunks for answering. This "small-to-big" retrieval is the workhorse pattern for long documents, and it directly solves the tension — precision and context stop being a single dial you have to compromise on.
Summary nodes for whole-document questions
Parent-child fixes local context, but long documents also invite global questions: "What is this contract's overall risk posture?" "What are the main themes of this paper?" No single passage answers those — they're about the document as a whole, and passage retrieval will only ever surface fragments.
Summary nodes address this. You build a hierarchy of summaries: summarize each section, then summarize the summaries, up to a document-level summary. These summaries are themselves chunked and embedded, so they're retrievable. Now:
- A specific question retrieves a leaf passage (with parent context).
- A broad question retrieves a summary node — the section or document summary that actually addresses a holistic query.
The hierarchy lets retrieval match the granularity of the question: detailed questions hit detailed chunks, big-picture questions hit summaries. A flat index can only ever return passages, so it answers big-picture questions poorly; a hierarchy gives them the right unit to retrieve.
The costs
Hierarchy isn't free, and it's worth being honest about the overhead:
- Heavier ingestion. You're producing multiple chunk levels and generating summaries (usually via an LLM), which is more compute and more moving parts than flat chunking. Summary generation in particular adds cost and a potential error source — a bad summary misleads retrieval.
- More storage and complexity. Multiple granularities mean more vectors and relationships to manage, plus retrieval logic that knows how to walk parent-child links and route to summaries.
- Maintenance. When the document changes, summaries and parents may need regenerating, not just the changed leaf.
When it's worth it
Hierarchical chunking earns its complexity specifically on long, structured, dependency-heavy documents:
- Legal contracts (clauses depend on definitions and each other)
- Research papers and technical reports (sections build on each other)
- Books, manuals, and long-form documentation
- Any corpus where users ask both pinpoint questions and whole-document questions
For short, self-contained content — FAQs, support articles, most web pages — flat chunking with good overlap is simpler and just as effective; a hierarchy is over-engineering there.
The mental model: flat chunking assumes documents are bags of independent passages. Long documents aren't — they're trees with long-range dependencies and global meaning. Hierarchical chunking makes the chunk structure match the document structure, so you can retrieve a precise passage without severing it from the context that makes it mean anything, and answer whole-document questions with a unit that actually spans the document. Match the chunking model to the shape of your content, and long documents stop being the place your RAG system quietly falls apart.