Chunking strategies for RAG: fixed-size, semantic, and recursive

The quality of your retrieval is upstream of the quality of your chunks.

Why chunking matters

Large language models have context windows. When a document is too long to fit, you split it into chunks, embed each chunk, and retrieve the most relevant ones at query time. The way you split determines what each embedding represents, and therefore what your model can find.

Strategy 1: Fixed-size chunking

Split every N tokens, optionally with a K-token overlap.

def fixed_chunk(text: str, size: int = 512, overlap: int = 64) -> list[str]:
    tokens = tokenize(text)
    return [detokenize(tokens[i:i+size]) for i in range(0, len(tokens), size - overlap)]

Pros: simple, predictable, fast
Cons: splits mid-sentence, mid-concept; overlap wastes embedding budget

Strategy 2: Recursive character splitting

Split on paragraph → sentence → word boundaries, stopping when the chunk is small enough.

This preserves natural language boundaries while still keeping chunks within a size budget. It is the default in most RAG frameworks.

Strategy 3: Semantic chunking

Embed each sentence, then split where consecutive sentence similarity drops below a threshold.

def semantic_chunk(sentences, threshold=0.7):
    chunks, current = [], [sentences[0]]
    for prev, curr in zip(sentences, sentences[1:]):
        if cosine_similarity(embed(prev), embed(curr)) < threshold:
            chunks.append(" ".join(current))
            current = []
        current.append(curr)
    chunks.append(" ".join(current))
    return chunks

Pros: chunks align with topic shifts
Cons: requires an extra embedding pass at ingestion time

Benchmark results

We tested all three strategies against five document types (legal contracts, API docs, news articles, product manuals, Q&A threads) using recall@5 as the metric, with a single general-purpose embedding model and a shared evaluation set of ~200 queries per document type.

Recursive splitting wins on structured reference content where natural boundaries are already clear. Semantic chunking can outperform it on long-form documents with dense topical shifts, though the gap narrows on shorter or well-structured documents, and it costs roughly 14× more compute at ingestion time. Independent benchmarks show mixed results: gains are real in some corpora and negligible in others.

Recommendation

Start with recursive splitting — it handles most content types well, is fast, and needs no extra embedding pass at ingestion. Move to semantic chunking only if you have dense, topic-shifting documents (legal contracts, research papers, Q&A threads) and your evaluation set shows a meaningful recall improvement that justifies the ingestion overhead.