Kognita
All posts
Engineering2026-08-10·8 min read

CoRAG explained: breaking queries into sub-questions for better answers

Some questions can't be answered by one retrieval. Chain-of-retrieval augmented generation decomposes a hard query into a sequence of sub-questions, retrieving and reasoning at each step.

Single-shot RAG works because most questions map cleanly to a passage: ask one thing, retrieve the relevant chunk, answer. But a large class of questions doesn't work that way. "Which of our enterprise customers signed after the pricing change and are up for renewal next quarter?" isn't in any single document. Answering it requires finding the pricing change date, then the customers who signed after it, then cross-referencing renewal dates. One retrieval can't do that. CoRAG — chain-of-retrieval augmented generation — is the pattern for questions that need a chain of retrievals, not one.

CoRAG breaking queries into sub-questions

Why single-shot retrieval falls short

A standard RAG query embeds the whole question and retrieves the nearest passages. That works when the answer lives together in the corpus. It breaks on:

  • Multi-hop questions where the answer depends on an intermediate fact you have to find first ("who wrote the doc that the onboarding runbook references?").
  • Compositional questions that combine several independently-retrievable facts ("compare our refund policy to our SLA credit policy").
  • Questions with an implicit prerequisite the user didn't state, but the system has to resolve to answer correctly.

Embed one of these as a single vector and you get a muddled query that's near everything and precisely on nothing. The retriever can't serve a question whose pieces live in different places.

The CoRAG loop

CoRAG makes retrieval iterative. Instead of one retrieve-then-answer, it runs a loop:

  1. Decompose — the model breaks the question into the next sub-question it needs answered.
  2. Retrieve — run retrieval for that focused sub-question.
  3. Reason — incorporate the retrieved passage; decide whether the original question is now answerable.
  4. Repeat or answer — if not, generate the next sub-question (informed by what it just learned) and loop; otherwise synthesize the final answer.
Q: "Which customers signed after the pricing change are up for renewal?"
  → sub-Q1: "When did the pricing change happen?"      → retrieve → "March 2026"
  → sub-Q2: "Which customers signed after March 2026?" → retrieve → [list]
  → sub-Q3: "Which of those renew next quarter?"       → retrieve → [subset]
  → synthesize final answer

The key property: each sub-question is focused enough for retrieval to succeed, and later sub-questions are shaped by earlier answers. The chain accumulates the facts a single query could never gather.

Why decomposition helps retrieval specifically

There's a retrieval-quality reason CoRAG works beyond just "breaking down hard problems." A focused sub-question is a better query than a compound one. "When did the pricing change?" embeds to a tight point near the passage that states the date. The original compound question embeds to a blurry average of several topics. Decomposition doesn't just organize reasoning — it produces queries the retriever can actually satisfy, which is where single-shot RAG was failing.

The costs

CoRAG isn't free, and the costs are real:

  • Latency multiplies. Each hop is a retrieval plus a model call, in sequence. A three-hop chain is roughly three times the latency of single-shot. This is inherently serial — you can't retrieve sub-question 2 until you've answered sub-question 1.
  • Errors compound. A wrong intermediate answer sends the whole chain down a bad path. Later hops reason from an incorrect fact and confidently produce a wrong final answer. Chains need guardrails — grading intermediate results, capping hop count — or they wander.
  • Evaluation is harder. The same question can take different chains on different runs, so debugging means tracing the whole sequence, not one retrieval.

When to reach for it

CoRAG is a targeted tool, not a default. Use it when:

  • Your questions are genuinely multi-hop or compositional, and you can see single-shot RAG failing because the pieces live apart.
  • The value of the answer justifies the latency — an analyst's complex query can afford a few seconds; a real-time chat autocomplete cannot.

For the majority of FAQ-style and single-fact questions, single-shot retrieval with good hybrid search and reranking is faster and just as accurate. Reach for CoRAG when you've confirmed the failure is structural — the question needs a chain because its answer is spread across a chain of facts. Applied there, it turns unanswerable queries into answerable ones. Applied everywhere, it's latency you didn't need to spend.