Measure retrieval before you tune prompts
· 8 min read
When a RAG system gives a wrong answer, the first thing people reach for is the prompt. Add "only use the context below", add "if you don't know, say so", try another model. Sometimes it helps. Often the real problem happened one step earlier: the chunk with the answer never reached the prompt, and no wording can make a model quote a document it was never shown.
So I split evaluation into two questions and answer them in order. Did retrieval find the right sources? Given those sources, did generation use them well?
