Every team building RAG eventually has the same bad day: a change that "improved" retrieval quietly broke it somewhere else, and nobody noticed until users complained. The fix is the same discipline you already apply to code — automated tests. A RAG evaluation suite is a regression harness for answer quality: a fixed set of questions with expected results, scored automatically, run on every change. Without it, every tweak is a gamble. With it, you know whether a change helped or hurt before it ships. Here's how to build one.
Why RAG needs tests specifically
RAG systems are unusually easy to break invisibly. A change to chunk size, embedding model, retrieval parameters, or the prompt can improve some queries and regress others, with no error to signal the regression — retrieval just silently gets worse on a subset. And the surface area is large: chunking, embedding, retrieval, ranking, and generation each affect quality and interact. Manual spot-checking ("I tried a few questions, seems fine") samples a tiny, biased slice and misses regressions elsewhere. Automated evals are the only way to know a change's net effect across the range of questions you care about.
Step 1: Build a golden dataset
The foundation is a golden dataset — representative questions paired with known-correct answers, and ideally the specific passages that should be retrieved. This is your ground truth, and its quality caps the value of everything else.
- Make it representative. Cover the real distribution of questions — common ones, edge cases, different phrasings, questions that should have no answer (to test that the system declines rather than fabricates). A dataset of only easy questions gives false confidence.
- Include the expected passages where you can. Recording which chunks should be retrieved lets you measure retrieval directly, without an LLM judge.
- Size it sensibly. Even 50–100 well-chosen pairs beat ad-hoc testing. Grow it over time — every real bug becomes a new test case so it can never regress unnoticed.
- Source it from reality. Mine real user queries (support tickets, search logs), including the ones your system currently fails. Real questions beat invented ones.
Step 2: Measure retrieval and generation separately
A RAG answer fails for two different reasons — bad retrieval or bad generation — and they need separate scoring, because the fixes are opposite. Measure both:
Retrieval metrics (if your golden set records expected passages, you can compute these directly, no LLM needed):
- Context recall — did retrieval find the passages needed to answer? This is the ceiling on everything.
- Context precision — how much of what was retrieved is actually relevant?
Generation metrics (usually scored with an LLM-as-judge, given question + context + answer):
- Faithfulness — is the answer grounded in the retrieved context, or does it fabricate/over-reach?
- Answer relevancy — does the answer actually address the question?
Keeping these separate is what makes an eval diagnostic: when a change moves a number, you see which stage it affected. Recall dropped? Retrieval regressed. Faithfulness dropped but recall held? The prompt or model is the culprit.
Step 3: Automate the scoring
For fuzzy metrics (faithfulness, relevancy), use an LLM-as-judge: a strong model scores each answer against the question and context. Frameworks like RAGAS package exactly this, or you can write your own judge prompts. Two cautions:
- Calibrate the judge. Check its scores against a handful of human-labeled examples so you trust it. An uncalibrated judge produces confident-but-meaningless numbers.
- Use direct metrics where possible. Retrieval recall/precision against your golden passages needs no LLM — it's exact set comparison. Prefer deterministic scoring where you have ground truth; reserve the LLM judge for what genuinely requires judgment.
Step 4: Run it like a test suite
The whole point is to make evals part of your change workflow, not a once-a-quarter exercise:
- Establish a baseline. Run the suite on your current system and record the numbers. That's the bar every change must clear.
- Run on every meaningful change — new chunk size, embedding model, reranker, prompt, retrieval params. Compare against baseline.
- Gate on regressions. Treat a significant drop in a key metric like a failing test: it blocks the change until understood. This is what actually prevents shipping regressions.
- Track over time. Watch the trend, so slow drift (e.g. from a growing, messier corpus) shows up before it becomes a user complaint.
What good looks like
A mature RAG eval setup looks a lot like a CI test suite: a versioned golden dataset, a scoring harness that reports retrieval and generation metrics, a baseline, and a gate that flags regressions on every change. When someone proposes "let's switch the embedding model," you don't debate it on intuition — you run the suite and read the delta. When context recall jumps but faithfulness dips, you see the trade-off and decide deliberately.
The mindset shift is the whole message: RAG quality is testable, and untested RAG changes are how regressions reach production. You wouldn't merge code without tests because you know you'd break things invisibly. RAG is exactly the same — a system with lots of interacting parts and failures that don't throw errors. Build the golden dataset, measure retrieval and generation separately, automate the scoring, and gate changes on it. It's the difference between improving your knowledge base with confidence and hoping each tweak didn't quietly make it worse.