Teams spend weeks debating GPT vs. Claude vs. the latest open-weights model, then ship a RAG system that still hallucinates. The reflex is to blame the LLM. Usually, the LLM is the one part that was working.
Generation quality has a ceiling you already hit
Modern LLMs are extraordinarily good at one thing that matters here: taking relevant text and turning it into a fluent, grounded answer. Give a frontier model the correct passage and a clear question, and it will almost always produce a correct answer. The marginal difference between the top few models on this specific task, summarising provided context, is small.
What LLMs cannot do is answer from a document they never received. If retrieval hands the model three irrelevant chunks, the model has two options: refuse, or invent. Both look like "the model is bad." Neither is a generation problem.
Where recall actually leaks
A RAG pipeline is a chain, and end-to-end quality is the product of each stage, not the maximum. Context is lost at every hop:
Document → chunking → embedding → indexing → retrieval → ranking → prompt
- Chunking splits a coherent answer across two chunks, so neither is a strong match on its own.
- Embedding collapses a rare identifier or negation into a nearby-but-wrong region of vector space.
- Retrieval returns top-k by similarity, but the answer sits at rank 12 and your k is 5.
- Ranking orders candidates so the best passage is present but buried below the model's attention.
If each stage is "90% good," four stages multiply to about 65%. The generation step then operates on whatever survived and cannot recover what was dropped upstream.
The diagnostic that settles the argument
Before touching the model, run this test. Take 50 questions you know the answers to. For each, log the chunks retrieval returned and ask a simple question:
Was the passage containing the answer anywhere in the retrieved set?
This is context recall, and it decomposes your problem cleanly:
| Passage retrieved? | Answer correct? | Diagnosis |
|---|---|---|
| No | No | Retrieval problem: fix this first |
| Yes | No | Generation or prompting problem |
| Yes | Yes | Working as intended |
In practice, the top-left row dominates. Most "the AI gave a wrong answer" tickets trace back to the correct passage simply not being in the context window. You cannot prompt-engineer your way out of missing information.
What to fix, in order
Once you know retrieval is the constraint, the levers are well understood and cheap relative to swapping models:
- Hybrid search. Pure vector search misses exact tokens: error codes, SKUs, proper nouns. Adding BM25 alongside embeddings typically lifts recall 20–30% with no model change.
- Reranking. Retrieve a wide candidate set (k=50), then use a cross-encoder to re-score and keep the top 5. This raises precision without sacrificing recall.
- Better chunking. Align chunk boundaries with topic boundaries so a complete answer lives in one chunk. Add overlap so context that straddles a boundary isn't orphaned.
- Increase k thoughtfully. More candidates raise recall but dilute the prompt. Reranking is what lets you retrieve wide and prompt narrow.
Notice what is not on this list: changing the LLM. That comes last, after the right documents are reliably reaching it.
The mindset shift
RAG stands for retrieval-augmented generation, and the field spent its first years optimising the wrong half of the acronym. Treat retrieval as the primary system you are building and generation as the thin, reliable layer on top. Measure context recall before you measure answer quality. When answers are wrong, assume the document didn't arrive until you've proven it did.
The best model in the world is only as good as the passage you put in front of it.