"Our search got better" is a claim, not a measurement. To actually know whether a change helped, whether from a new embedding model, a reranker, or a chunking tweak, you need numbers. This is a tour of the four retrieval metrics you'll reach for most, what each rewards, and, crucially, when to use which.
First: you need ground truth
Every metric here compares your system's ranked results against a set of known-correct answers. So before any measurement, you need an evaluation set: a list of queries, each paired with the document(s) that should be retrieved. Fifty to a few hundred realistic queries is enough to start. Without this, you're guessing.
Run each query through your system, get back a ranked list, and score how well the known-correct documents show up. That scoring is what these metrics formalise.
Hit Rate @ k: "did we get it at all?"
The simplest and most fundamental. For each query, ask: is the correct document anywhere in the top k results? It's 1 if yes, 0 if no; average across all queries.
Hit Rate@5 = (queries where the answer appears in top 5) / (total queries)
- Rewards: recall. Did the right document make it into the candidate set at all?
- Ignores: where it landed. Rank 1 and rank 5 score identically.
Use it when: something downstream re-reads all k results, most importantly RAG. If you feed the top 5 chunks to an LLM, you mostly care that the right chunk is in those 5; the model attends to all of them. Hit Rate is your north-star metric for retrieval feeding generation.
MRR: "how high was the first right answer?"
Mean Reciprocal Rank cares about the position of the first correct result. If it's at rank 1 you score 1; rank 2 scores 1/2; rank 3 scores 1/3, and so on. Average the reciprocals across queries.
MRR = mean( 1 / rank_of_first_correct_result )
- Rewards: getting a correct answer to the very top.
- Ignores: everything after the first hit. If there are three relevant docs, MRR only looks at the first one it finds.
Use it when: there's a single right answer and users look at the top result, as in "I'm feeling lucky" scenarios. Question-answering where one passage suffices, or a jump-to-the-doc search box. MRR punishes a system that buries the answer at rank 4 even if it technically retrieved it.
NDCG: "is the whole ranking well-ordered?"
Normalized Discounted Cumulative Gain is the most sophisticated of the four. It handles two things the others don't: graded relevance (results can be "perfect," "okay," or "irrelevant," not just right/wrong) and position discounting (a great result at rank 1 is worth more than the same result at rank 8). It then normalises against the best possible ordering, so 1.0 means "perfectly ranked."
- Rewards: putting the most relevant results highest, across the entire ranked list.
- Ignores: nothing much, which is also its cost: it needs graded relevance labels, which are more work to produce.
Use it when: users scan a whole ranked list and order matters throughout, as in classic search results pages, e-commerce, and anything where "results 1 through 10, best first" is the experience. NDCG is the right metric when the ranking itself is the product.
Precision & Recall @ k: the honest baseline
Worth keeping in your head as the two forces every ranking trades off:
- Recall@k: of all the relevant documents, how many appeared in the top k? (Did we miss anything?)
- Precision@k: of the top k results, how many were actually relevant? (Is the list full of junk?)
Push k up and recall rises but precision usually falls. This tension is exactly why retrieve-wide-then-rerank works: retrieve a large k for recall, then a reranker restores precision by re-ordering the best to the top.
Choosing the right one
| Your situation | Reach for |
|---|---|
| Feeding top-k chunks to an LLM (RAG) | Hit Rate@k |
| One right answer, users read the top result | MRR |
| A ranked results page users scan top-to-bottom | NDCG |
| Diagnosing recall vs. noise trade-offs | Precision & Recall@k |
A practical setup: track Hit Rate@k as your headline number if you're doing RAG, watch MRR to make sure the best result isn't drifting down the list, and pull out NDCG when you're tuning a reranker and ordering genuinely matters.
The metric that beats all of them
One caution. These metrics measure retrieval, not answer quality. A system can post a great Hit Rate and still produce bad final answers if generation or prompting is weak. So pair retrieval metrics with an end-to-end answer evaluation.
But, and this is the point of measuring retrieval separately, when the final answer is wrong, these metrics tell you where to look. Low Hit Rate? The document never arrived; fix retrieval. High Hit Rate but wrong answers? Retrieval did its job; look at generation. That decomposition is why you measure the layers independently instead of only judging the final output.
You can't improve what you don't measure. Start with an evaluation set, pick the metric that matches how your results are consumed, and let the numbers, not just gut feelings, tell you whether your last change actually helped.