Most teams ship RAG improvements on vibes. Someone tweaks the chunk size, tries a few queries, decides it "feels better," and merges. Then quality regresses somewhere else and nobody notices for weeks. The fix is to treat RAG like any other system with measurable behavior: decompose quality into metrics, measure them against a fixed evaluation set, and know which part moved when a number changes. Here are the five metrics that matter, split across the two things a RAG system does — retrieve and generate.
First, split the problem in two
A RAG answer can be wrong for two fundamentally different reasons: retrieval handed the model the wrong context, or generation mishandled the right context. These need separate metrics, because the fixes are opposite. Improving your prompt does nothing if retrieval never surfaced the answer; improving retrieval does nothing if the model ignores good context. Measure them apart, or you'll tune the wrong half.
Retrieval metrics
1. Context recall — did retrieval find the answer at all?
Of the information needed to answer correctly, how much made it into the retrieved context? This is the ceiling on your whole system. If the answer wasn't retrieved, no model can produce it — recall is the single most important retrieval metric because it bounds everything downstream. Low recall means your retriever (or chunking, or embedding) is losing the answer before generation even starts.
2. Context precision — is the retrieved context mostly signal?
Of what you retrieved, how much is actually relevant? High recall with low precision means you're burying the answer in noise — the model has to find the needle among distractor passages, which hurts answer quality and wastes context budget. Precision is what reranking primarily improves: same recall, less junk in the top-k.
The recall/precision split is diagnostic. Low recall → fix retrieval reach (hybrid search, query expansion, chunking). High recall but low precision → fix ranking (add or improve reranking).
Generation metrics
3. Faithfulness — is the answer grounded in the retrieved context?
Does every claim in the answer actually follow from the passages provided? This is the direct measure of hallucination-by-fabrication. A faithful answer says only what the context supports. An unfaithful one blends in the model's training-data recall or over-extrapolates. Faithfulness isolates whether the model is staying grounded — independent of whether the context was any good.
4. Answer relevancy — does the answer address the question?
An answer can be perfectly faithful to the context and still not answer what the user asked — it retrieved and grounded correctly, but on a tangent. Answer relevancy measures whether the response is on-target for the actual query. Low relevancy with high faithfulness often points to a retrieval-precision problem feeding the model the wrong (but real) context.
5. Hallucination rate — the bottom-line trust metric.
The headline number: how often does the system state something false or unsupported? It's downstream of faithfulness and context quality, but it's the one stakeholders care about, and it's what determines whether users trust the system. Track it as your north-star, and use the other four to explain why it moved.
How to actually measure these
Metrics need a fixed target to measure against:
- Build a golden dataset. A set of representative questions with known-correct answers and, ideally, the passages that should be retrieved. This is your regression harness — without it, every "improvement" is anecdote. Even 50–100 well-chosen Q&A pairs beats ad-hoc testing.
- Use LLM-as-judge for the fuzzy metrics. Faithfulness and answer relevancy are hard to score with string matching. A strong model, given the question, context, and answer, can grade them reasonably — frameworks like RAGAS automate exactly this. Calibrate the judge against a few human-labeled examples so you trust it.
- Measure retrieval separately with your golden passages. If you recorded which passages should be retrieved, you can compute recall and precision directly without an LLM judge at all.
The workflow this enables
Once these five are wired into a repeatable eval, the whole loop changes. You make a change — new chunk size, a reranker, a different embedding model — and you see which metric moved. Context recall jumped but faithfulness dropped? You surfaced more but the model is now over-reaching. Precision up, relevancy flat? The reranker helped but retrieval was already finding the right thing. You stop guessing and start doing engineering.
The point isn't the specific framework. It's that "it feels better" is not a measurement, and RAG quality is measurable if you split it into retrieval and generation and track these five. A team that measures ships improvements that stick; a team that vibes ships regressions it won't notice until a user does.