Measure retrieval before you tune prompts
When a RAG system gives a wrong answer, the first thing people reach for is the prompt. Add "only use the context below", add "if you don't know, say so", try another model. Sometimes it helps. Often the real problem happened one step earlier: the chunk with the answer never reached the prompt, and no wording can make a model quote a document it was never shown.
So I split evaluation into two questions and answer them in order. Did retrieval find the right sources? Given those sources, did generation use them well?
Two problems, two kinds of measurement
Retrieval is a search problem, and search has had good metrics for decades. You need a list of questions and, for each one, the sources that contain the answer. Run the retriever, compare. It needs no LLM, it is cheap, and with a fixed index it gives the same number every time.
Generation is harder to score. The answer is free text, and "correct" depends on what the sources say. The usual measures are:
- Faithfulness, or groundedness: is every claim in the answer supported by the retrieved context? This catches the model filling gaps from its own training.
- Answer relevance: does the answer address the question that was asked, or a nearby one?
Both are usually scored by another model, which brings its own problems (more on that below). That is another reason to start with retrieval: it is the half you can measure exactly.
Two retrieval metrics are enough
Recall@k asks: of the sources that contain the answer, how many are in the top k results? If a
question needs two documents and the top five contain one of them, recall@5 for that question is 0.5.
Average it over all questions. Pick k to match what you actually put in the prompt: if five chunks go
in, recall@5 is the number that matters, and recall@50 tells you whether a reranker has something to work
with.
MRR, mean reciprocal rank, asks how high the first right source is. A question whose first relevant result is in position 1 scores 1, position 2 scores 0.5, position 4 scores 0.25, and a miss scores 0. Recall tells you whether the answer is there at all; MRR tells you whether it is near the top, where the model pays the most attention.
Both are a few lines of C#:
public sealed record GoldenCase(string Question, IReadOnlySet<string> ExpectedSources);
public static class RetrievalMetrics
{
// Share of the expected sources found in the first k results.
public static double RecallAtK(IReadOnlyList<string> retrieved, IReadOnlySet<string> expected, int k) =>
(double)retrieved.Take(k).Distinct().Count(expected.Contains) / expected.Count;
// 1 / position of the first expected source; 0 if none was retrieved.
public static double ReciprocalRank(IReadOnlyList<string> retrieved, IReadOnlySet<string> expected)
{
for (int i = 0; i < retrieved.Count; i++)
{
if (expected.Contains(retrieved[i])) return 1.0 / (i + 1);
}
return 0;
}
}
And the loop that runs them over the whole set, given a function that returns the source IDs of the top results in order:
public static async Task<(double RecallAtK, double Mrr)> EvaluateAsync(
IReadOnlyList<GoldenCase> cases,
Func<string, Task<IReadOnlyList<string>>> search,
int k)
{
double recall = 0, rr = 0;
foreach (var c in cases)
{
IReadOnlyList<string> sources = await search(c.Question);
recall += RetrievalMetrics.RecallAtK(sources, c.ExpectedSources, k);
rr += RetrievalMetrics.ReciprocalRank(sources, c.ExpectedSources);
}
return (recall / cases.Count, rr / cases.Count);
}
One detail decides whether this survives its first refactor: label sources, not chunks. If the
expected answer is "chunk 1834", the labels are invalid the moment you change the chunk size. If it is
"the refunds runbook" (or the runbook plus a section heading), map each retrieved chunk to its source
document and the same golden set keeps working through any chunking change. That is also why the code
calls Distinct(): three chunks from the same document count once.
A golden set of 30 to 50 questions
The golden set doesn't need to be large. It needs to be real and to cover the ways retrieval fails. I would build it like this:
- Start from real questions where they exist: support tickets, search logs, what people ask in chat. Questions written by the person who built the index are phrased like the documents and flatter it.
- Include exact-term questions: error codes, product names, IDs, version numbers. These are where pure vector search fails, and you want to see it.
- Include questions that need two sources, so recall below 1 means something.
- Include a few questions the corpus can't answer. They have no expected source, so they don't count towards retrieval metrics, but they are the best test of whether generation admits it doesn't know.
- Label by hand. Someone who knows the documents writes down which ones contain the answer. This is the expensive part, an afternoon for 40 questions, and it is the part nobody can do for you.
Keep it in the repository as a plain file, next to the code it tests:
[
{ "question": "What does ERR_4012 mean during checkout?", "expectedSources": ["runbook-payments"] },
{ "question": "Can I get a refund after 30 days?", "expectedSources": ["policy-refunds", "faq-billing"] }
]
Be honest about what a small set can tell you. With 40 questions, one question is 2.5 points of recall. A change of one or two questions is noise; look at which questions flipped, not only at the average. The point of the set is to catch regressions you would otherwise ship, not to put a number on a slide.
Run it in CI
Retrieval evaluation is cheap enough to run on every change that can affect it:
- chunk size and overlap,
- the embedding model, or its version,
k, the candidate list size, the fusion settings,- adding or changing a reranker,
- changes to how documents are cleaned before indexing.
Commit the current numbers as a baseline, and fail the build when recall@k or MRR drops by more than a margin you chose on purpose. Print the questions that went from hit to miss; that list is more useful than the totals. When a change is meant to move the numbers, update the baseline in the same pull request, so the review shows what it cost and what it bought.
Evaluate against a fixed snapshot of the documents. If the corpus changes between two runs, you are no longer comparing two retrievers. And when you change the embedding model, the snapshot has to be re-embedded with it; comparing new query embeddings against an old index tells you nothing.
Generation, and the judge problem
Once retrieval holds steady, it is worth scoring the answers. Faithfulness and relevance are usually
scored by an LLM acting as a judge: it gets the question, the retrieved context and the answer, and
returns a score with a reason. In .NET, the Microsoft.Extensions.AI.Evaluation.Quality package ships
evaluators of this kind, among them GroundednessEvaluator, RelevanceEvaluator and
RetrievalEvaluator, each scoring 1 to 5 with a model you configure. The companion Reporting package
caches the model's responses and produces reports, which matters when the same set runs again and again.
A judge is useful, but it is a measurement instrument with known flaws:
- It isn't deterministic. The same answer can get a 4 today and a 3 tomorrow, even at temperature 0.
- It has biases. Judges tend to prefer longer answers, and a model judging its own output tends to like it.
- It changes with its model. Upgrade the judge and every score moves; pin its version and treat a judge change like any other change to the test.
- It costs money and time on every run.
So I would use judge scores for trends, not as truth. Grade ten or twenty answers by hand first and check that the judge mostly agrees with you; if it doesn't, fix the judge before trusting its numbers. Run generation evaluation when prompts or models change, or on a schedule, rather than on every commit.
Why this order
If recall@5 is 0.6, four in ten questions reach the model without the source that answers them. No prompt fixes that, and every hour spent tuning the prompt is measured against a broken input. Fix retrieval until the numbers stop moving, then tune the prompt with a judge watching faithfulness. The work is the same either way; the order decides whether you can tell what helped.
If you are adding hybrid search or a reranker, Vectors are not enough covers those, and this golden set is how to tell whether they paid off.
