Kognita
All posts
Engineering2026-07-30·9 min read

How to debug a retrieval pipeline: finding the step that's losing context

"The bot gave a bad answer" is a symptom, not a diagnosis. A RAG pipeline has four places context can leak, and each fails differently. Here's how to isolate which one broke.

When a RAG system gives a wrong answer, the instinct is to blame the model. Usually the model is innocent. A RAG pipeline is a chain — chunking → embedding → retrieval → ranking → generation — and a failure at any early link produces a bad answer at the end, with the model faithfully working from context that was already broken before it arrived. Debugging RAG is the discipline of finding which link leaked, instead of tuning the last one and hoping.

Here's how to isolate the failing step, working backward from the symptom.

debugging a retrieval pipeline

The golden rule: check the context before blaming the model

The single most useful debugging habit: for any bad answer, look at the passages that were actually retrieved and handed to the model. This one check splits every failure into two worlds:

  • The right passage was in the context, and the model still answered wrong. Now it's a generation problem — prompt, or model.
  • The right passage was not in the context. The model never had a chance. It's a retrieval problem, and no amount of prompt tuning will fix it.

Most teams skip this and go straight to fiddling with the prompt. If the answer wasn't in the retrieved context, the prompt was never the issue. Always look at the context first.

Isolating the failing step

Once you know it's a retrieval problem, walk the chain backward. Each step fails in a recognizable way.

Step 4 — Ranking: right document, wrong position

The most common failure. The correct passage was retrieved but ranked low — position 8 when you only pass the top 5 to the model. Test it directly: pull a wider result set (top 20) and check whether the answer is in there but below your cutoff.

  • Symptom: the answer is in the candidate set but not in the top-k you pass to the model.
  • Fix: add or improve reranking. This is exactly the "right doc, wrong rank" problem a cross-encoder reranker exists to solve.

Step 3 — Retrieval: the passage isn't in the candidate set at all

Pull the top 50 and the correct passage still isn't there. The retriever couldn't find it. Two usual causes:

  • Vocabulary mismatch — the query and answer share no words, and pure vector search couldn't bridge it. Test by adding keyword search: if hybrid retrieval finds it, the semantic match was the gap. Query expansion or HyDE also target this.

  • The exact-term case — the query hinges on an error code or proper noun that vector search treats as meaningless. Hybrid search is the fix.

  • Symptom: the answer exists in your corpus but never appears in retrieval results, even a wide pull.

  • Fix: hybrid search; query reformulation; check the embedding model is appropriate for your domain.

Step 2 — Embedding: the meaning didn't survive

Retrieval can't find what was never embedded meaningfully. Check whether the content was embedded at all, and with a model suited to your domain. Highly technical or specialized content sometimes needs a domain-appropriate embedding model; a generic one may collapse distinctions that matter.

  • Symptom: even direct, near-verbatim queries fail to retrieve the passage.
  • Fix: confirm the content actually embedded; evaluate whether the embedding model fits your content type.

Step 1 — Chunking: the answer was destroyed before embedding

The subtlest failure, and the one people check last. If chunking split the answer across a boundary, no chunk contains the complete thought — so no embedding represents it, so retrieval literally cannot return it. Go look at the raw chunks for the source document. Is the answer intact in a single chunk, or bisected across two?

  • Symptom: the source document clearly contains the answer, but the chunks each contain only a fragment of it.
  • Fix: increase chunk overlap so boundary-straddling answers survive; reconsider chunk size for your content.

A debugging checklist

Run these in order and you'll localize almost any retrieval failure:

  1. Look at the retrieved context. Was the answer in it? No → retrieval problem (continue). Yes → generation problem (stop; fix the prompt or model).
  2. Widen the result set to top 20–50. Is the answer there now but below your cutoff? → ranking problem. Add reranking.
  3. Add keyword/hybrid search. Does the answer appear now? → retrieval / semantic-match problem.
  4. Inspect the raw chunks of the source document. Is the answer split across a boundary? → chunking problem. Increase overlap.
  5. Confirm the content embedded and the model fits the domain. → embedding problem.

Instrument for this in advance

You can only debug what you can see. Build the pipeline so that for any answer you can inspect: the exact chunks of each document, the retrieval candidate set with scores, the post-rerank ordering, and the final context sent to the model. Kognita surfaces the retrieved passages and their sources with every search result precisely so this inspection is possible — when an answer looks wrong, you can trace it back to which passages fed it and where in the ranking they sat, rather than guessing.

The meta-lesson: "bad answer" is never a diagnosis. It's the last event in a chain, and the fault is usually upstream of where it shows up. Resist tuning the model. Walk the chain backward, look at the context first, and fix the step that actually leaked.