Every RAG evaluation rests on a golden dataset — the set of questions with known-correct answers (and ideally the passages that should be retrieved) that you score against. Get the dataset right and your evals catch regressions before users do. Get it wrong and you have a number that looks rigorous while measuring nothing useful, giving false confidence right up until production breaks. Building a good golden dataset is more craft than most teams expect. Here are ten lessons from doing it.
1. Source questions from reality, not imagination
The single biggest lever. Invented questions reflect what you think users ask; real questions reflect what they actually ask, which is messier, more varied, and full of phrasings you'd never guess. Mine support tickets, search logs (especially zero-result queries), and real user chats. A dataset built from real questions tests the system against its real workload.
2. Include the questions your system currently fails
It's tempting to fill the dataset with questions the system already handles well — and it produces a comforting high score that measures nothing. Deliberately include known failures and hard cases. The dataset's job is to expose weaknesses and catch regressions, not to flatter the current system. A dataset that only contains easy wins is a vanity metric.
3. Cover the distribution, including the tails
A good dataset spans the shape of real usage: common questions (the bulk of traffic), edge cases, ambiguous phrasings, and different ways of asking the same thing. Weight toward what's common, but don't omit the tails — edge cases are where regressions hide. Aim for representativeness, not just volume.
4. Record the expected passages, not just answers
Wherever you can, note which chunks should be retrieved for each question — not only the correct final answer. This is what lets you measure retrieval (recall/precision) directly and deterministically, separate from generation, without an LLM judge. It's extra work up front and it's the difference between "the answer was wrong" and "retrieval failed" — which are different bugs with different fixes.
5. Include "should refuse" cases
Add questions your system should not answer — things genuinely not in the corpus. The correct behavior is to decline ("I don't have that information"), not fabricate. Without these cases, your eval never tests the honesty behavior that's critical for trust, and you won't notice when a change makes the system start confidently guessing on out-of-scope questions.
6. Quality over quantity
A carefully curated 100-question set beats a sloppy 10,000-question one. Each entry should have a genuinely correct, agreed-upon answer — noisy ground truth produces noisy metrics you can't trust. Start small and high-quality; a tight dataset you believe in is more useful than a huge one you don't.
7. Get the ground truth right (it caps everything)
Your metrics can't be more reliable than your answer keys. If the "correct" answers are wrong, contradictory, or debatable, every score built on them is meaningless. Have someone knowledgeable validate the ground truth, resolve disagreements, and treat the answer key as something to get right, not just done. This is the least glamorous and most important step.
8. Grow it from every real bug
Make the dataset living. Every time a real failure surfaces in production, add it as a new test case with its correct answer. This does two things: it ensures that specific failure can never silently regress again, and it steadily grows the dataset toward the real distribution of hard cases. Your eval should get stronger every time something breaks.
9. Version it and keep it stable
Treat the golden dataset like code: version it, and keep it stable enough that scores are comparable across time. If the dataset churns constantly, you can't tell whether a metric moved because the system changed or the test changed. Add to it deliberately, and note when you do, so trend lines stay meaningful.
10. Keep it representative as your corpus evolves
Your content and your users' questions drift over time — new products, new topics, new phrasings. A golden dataset built two years ago may no longer reflect current usage. Periodically refresh it from recent real questions so it keeps testing what the system actually faces now, not what it faced at launch. A stale eval set measures a system that no longer exists.
The meta-lesson
A golden dataset is not a one-time deliverable you build, score once, and file away. It's a living asset that encodes what "good" means for your specific knowledge base — built from real questions, honest about failures, precise in its ground truth, and continuously grown from production reality. The teams whose RAG systems improve steadily are the ones who invested in this: they can make a change, run the suite, and know the net effect, because their ground truth is trustworthy and representative. The teams that ship regressions are usually the ones evaluating against a handful of easy invented questions — a dataset that says "all good" while users hit the cases it never contained. Build the dataset like the foundation it is, because every quality claim you make about your RAG system rests on it.