If you can only add one thing to a naive RAG pipeline, add a reranker. Nothing else — not a fancier embedding model, not semantic chunking, not query rewriting — reliably moves end-to-end answer quality as much for as little effort.
Here's why it works, and how to add it without wrecking your latency budget.
The problem with pure vector search
Vector search is a bi-encoder system. Your documents get embedded once at ingestion time into fixed vectors. At query time, the query gets embedded into its own vector, and you find the nearest document vectors by cosine similarity.
The catch is in the word once. Each document is compressed into a single vector with no knowledge of the query it'll eventually be matched against. All the nuance of "does this passage actually answer this specific question" has to survive being squeezed into one point in space, independently on both sides. That compression is what makes vector search fast — you precompute everything and the query is just a nearest-neighbor lookup — and it's also what makes it approximate. The right passage is usually in the top 20 results. It's just often not at position 1.
What a cross-encoder does differently
A cross-encoder doesn't precompute anything. It takes the query and one candidate passage together, as a single input, and outputs one number: how relevant this passage is to this query.
bi-encoder (vector search):
score = cosine( embed(query), embed(passage) ) # two separate passes
cross-encoder (reranker):
score = model( query, passage ) # one joint pass, reads both
Because it reads both sides at once, the cross-encoder can weigh the actual interaction between the words in the query and the words in the passage — negations, qualifiers, whether "Postgres connection pooling" is the topic or an aside. It's dramatically more accurate at judging relevance.
It's also dramatically more expensive: you can't precompute it, because the score depends on the query. You'd never run a cross-encoder over your whole corpus per query — that's millions of forward passes.
The two-stage pattern
So you don't. You use each model for what it's good at:
- Retrieve wide, cheap. Use vector (or hybrid) search to pull a generous candidate set — top 50 or top 100. This is fast and its only job is recall: make sure the right passage is somewhere in the pile.
- Rerank narrow, accurate. Run the cross-encoder over just those 50-100 candidates, scoring each against the query, and keep the top 5. This is the precision step.
candidates = hybrid_search(query, k=50) # fast, recall-oriented
scored = reranker.rank(query, candidates) # slow but only 50 items
context = scored[:5] # precise top-k for the prompt
You get the recall of a wide net and the precision of a model that actually reads. The cross-encoder only ever sees a few dozen passages, so the cost is bounded and predictable.
Why it's "the cheap trick"
- No re-indexing. Reranking sits entirely at query time. You don't touch your embeddings, your chunks, or your vector store. You can add it to an existing pipeline in an afternoon and remove it just as easily.
- It fixes the common failure directly. The most frequent RAG complaint is "it retrieved the right document but the answer used the wrong chunk." That's a ranking problem, and ranking is exactly what a reranker fixes.
- It compounds with everything else. A better embedding model improves your top-50. A reranker improves which 5 of those 50 the model reads. They stack.
The costs, honestly
It isn't free:
- Latency. Scoring 50 passages adds tens to a few hundred milliseconds depending on the model and whether you call a hosted reranker or run your own. Tune the candidate count — reranking 100 is rarely twice as good as reranking 30.
- A second dependency. You now have a reranking model in the path, whether that's a hosted API or a model you serve. It's one more thing that can be slow or down.
For most systems the trade is lopsidedly worth it: a large jump in the relevance of what the model reads, for a modest, tunable latency cost.
Where Kognita puts it
Reranking isn't an add-on you bolt on later in Kognita — hybrid retrieval feeds a reranking stage before results reach the model, because raw nearest-neighbor ranking is the wrong thing to hand an LLM. If you're running your own pipeline, the takeaway is the same: retrieve wide with something cheap, rerank narrow with something that reads, and give the model the top few. It's the highest quality-per-effort change available to you.