Ask a team building RAG what chunk size they use and you'll get a confident answer: 512 tokens, 256, 1000, whatever the tutorial said. Ask them about overlap and you'll usually get a shrug. That's backwards. Chunk size is a coarse dial that mostly trades precision against context. Overlap is what determines whether the sentence that answers the question survives the split at all.
The boundary problem
When you split a document into chunks, you draw hard lines through continuous text. Those lines don't care about meaning. Consider:
...The migration must run before the deploy. If it doesn't, the service will fail to start because the new columns won't exist yet...
Split at the wrong token and you get:
- Chunk A: "...The migration must run before the deploy. If it doesn't, the service will fail to start"
- Chunk B: "because the new columns won't exist yet. The rollback procedure is..."
Now a user asks why does the service fail to start? The answer — "because the new columns won't exist yet" — lives in Chunk B, but Chunk B opens with a dangling "because" and never mentions "service" or "start." Its embedding points somewhere unhelpful. The chunk that does mention the failure, Chunk A, got cut off right before the reason. The answer is in your corpus, correctly ingested, and structurally unretrievable.
No chunk size fixes this. A bigger chunk just moves the boundary somewhere else, where it splits a different idea.
What overlap does
Overlap means each chunk repeats the last N tokens of the previous one. With overlap, the boundary region appears in both chunks:
- Chunk A: "...the service will fail to start because the new columns won't exist yet."
- Chunk B: "...will fail to start because the new columns won't exist yet. The rollback procedure is..."
Now the complete cause-and-effect exists as a contiguous span in at least one chunk. Whichever way the query leans, there's a chunk that holds the whole thought. Overlap is cheap insurance against the guillotine landing in the worst possible spot.
def chunk_with_overlap(tokens, size=512, overlap=64):
step = size - overlap
return [tokens[i:i + size] for i in range(0, len(tokens), step)]
The one subtlety: your stride is size - overlap, not size. Bump overlap and you produce more chunks for the same document, which means more vectors to store and embed. That's the cost, and it's real but usually modest.
How much overlap
A useful rule of thumb is 10–20% of chunk size. For 512-token chunks, that's roughly 50–100 tokens of overlap — enough to carry a sentence or two across the boundary, which is where most split-answer damage happens.
- Too little (0–5%): you're back to the guillotine problem. Ideas that straddle a boundary get orphaned.
- Too much (40%+): heavy duplication. You store the same text many times, retrieval returns near-identical neighboring chunks that crowd out diversity, and your embedding bill climbs for little gain.
The sweet spot exists because the failure it prevents — an answer sentence bisected by a boundary — is a local phenomenon. It happens within a sentence or two of the cut. You don't need massive overlap to cover it; you need just enough to span the typical answer unit.
Why size gets the attention it doesn't deserve
Chunk size does matter — it sets the granularity of what you retrieve. Smaller chunks are more precise (the embedding represents one tight idea) but carry less surrounding context. Larger chunks carry more context but dilute the embedding across multiple ideas, hurting precision. That's a genuine trade-off worth tuning.
But it's a smooth trade-off. Move from 512 to 768 tokens and quality shifts gradually. Overlap failures are sharp: a specific answer is either intact or bisected, retrievable or not. Sharp failures on specific queries are what users actually notice and report, which is why overlap punches above its reputation.
Practical guidance
- Start with 512-token chunks and ~15% overlap as a sane default. It's roughly what Kognita's fixed-token strategy uses, and it's a good baseline before you optimize anything.
- If users report "the answer is in the docs but the bot can't find it," suspect a boundary problem before you suspect the embedding model. Look at where the source sentence sits relative to your chunk boundaries.
- Tune size against the shape of your content (dense reference vs. flowing prose). Tune overlap against the length of a typical answer — it needs to be long enough that a complete answer unit survives in one chunk.
Chunk size decides how the model sees your documents in general. Overlap decides whether one specific, boundary-straddling answer is reachable at all. The second failure is the one that generates support tickets.