How reranking models work: ColBERT, cross-encoders, and LLM-as-judge

Reranking is the single most reliable quality upgrade you can add to a RAG pipeline — retrieve a wide candidate set cheaply, then reorder it with a model that judges relevance far more accurately than vector similarity. But "add a reranker" hides real variety: cross-encoders, late-interaction models like ColBERT, and LLM-as-judge rerankers work very differently, with different speed/quality trade-offs. Understanding how each operates helps you pick the right one instead of treating "reranker" as a black box. Here's the ecosystem.

Why reranking exists: the bi-encoder limitation

First, the problem all rerankers solve. Vector search uses a bi-encoder: documents and queries are embedded separately into single vectors, and matched by similarity. That separate, precomputed encoding is what makes vector search fast — but it means each document is compressed into one vector with no knowledge of the query it'll face. Nuance gets lost in that compression, so the right passage is usually in the top-k candidates but often not ranked first.

Reranking fixes this by re-scoring a candidate set with a model that considers query and passage together (or more richly than a single dot product). It's always a second stage: cheap wide retrieval first (recall), precise reranking second (precision), over just a few dozen candidates so the cost is bounded. All the reranker families share this two-stage role; they differ in how they score.

Cross-encoders: read both together, output one score

The classic reranker. A cross-encoder takes the query and one candidate passage jointly, as a single input, and outputs one relevance score. Because it processes both together through the full model, it can weigh the actual interactions between query and passage words — negations, qualifiers, whether a term is the topic or an aside.

• Strength: very accurate. Reading both sides jointly is what makes it far better than bi-encoder similarity at judging true relevance.
• Cost: you can't precompute anything (the score depends on the query), so you run a full model pass per candidate. That's why it only works on a candidate set (top-50, not the whole corpus) — running it over millions of passages per query is infeasible.
• When: the default, workhorse reranker. Great accuracy on a bounded candidate set, the pattern most RAG systems (Kognita included) use.

ColBERT and late interaction: multi-vector matching

ColBERT (and late-interaction models generally) sit cleverly between bi-encoders and cross-encoders. Instead of compressing a passage into one vector (bi-encoder) or processing query+passage jointly every time (cross-encoder), late interaction encodes the query and passage into multiple vectors — one per token — and computes relevance via fine-grained token-level matching (each query token finds its best-matching passage token, summed up).

• Strength: more expressive than a single-vector bi-encoder (token-level matching captures nuance a single vector loses), while allowing passage token vectors to be precomputed — so it's faster than a cross-encoder at scale.
• Cost: storing multiple vectors per passage is more storage-heavy, and the matching is more complex than a dot product.
• When: you want better-than-bi-encoder quality with better-than-cross-encoder scalability — a middle ground, useful at larger scale or when you want richer first-stage retrieval, not just reranking.

LLM-as-judge: prompt a model to rank

The newest family: use a general LLM to rerank, by prompting it to score or order candidate passages by relevance to the query. The model reads the query and candidates and judges relevance with its full language understanding.

• Strength: maximally flexible and often very accurate — an LLM can apply nuanced, instruction-driven relevance criteria ("prefer passages that directly answer, not just mention the topic"). It can also explain why, which aids debugging.
• Cost: the most expensive and slowest option — full LLM inference per rerank, with real latency and token cost. Overkill for most high-volume, latency-sensitive retrieval.
• When: high-value, lower-volume scenarios where relevance judgment is subtle and worth the cost, or where you want explainable rankings. Also the mechanism behind LLM-based evaluation (scoring retrieval quality offline), where latency matters less.

Choosing among them

Practical guidance

• Start with a cross-encoder reranker. It's the standard for a reason: excellent accuracy on a bounded candidate set, straightforward to add, and it delivers the big "right doc, wrong rank → fixed" win. For most systems this is the answer.
• Consider late interaction (ColBERT-style) when you're at a scale where cross-encoder latency over candidates strains you, or you want richer retrieval quality with precomputable passage representations.
• Reserve LLM-as-judge for high-value, subtle-relevance cases where the cost is justified, and for offline evaluation where you're scoring quality rather than serving live traffic.
• Whatever you pick, keep the two-stage shape. Retrieve wide and cheap for recall, rerank narrow for precision. The reranker only ever sees a candidate set, which is what keeps its cost bounded.

The takeaway

"Add a reranker" is the best advice in RAG, but the reranker you add matters. Cross-encoders read query and passage together for accurate scoring and are the sensible default; ColBERT-style late interaction trades storage for token-level nuance that scales better; LLM-as-judge offers the most flexible, explainable ranking at the highest cost. They all play the same second-stage role — reordering a cheaply-retrieved candidate set — because they all exist to fix the same thing: the bi-encoder's habit of retrieving the right passage but ranking it fourth. Understand the mechanism, match it to your scale and stakes, and you turn "a reranker" from a black box into a deliberate choice.