Retrieval quality is a data problem, not a model problem
A better prompt won't fix retrieval that's returning the wrong passage.
When a RAG assistant gives a wrong or unsupported answer, the instinct is to blame the model — a better prompt, a bigger model, a stricter system message. Almost always, the actual fault is upstream: the retrieval step returned the wrong passage, or no passage at all, and the model is doing its best with bad source material.
Retrieval quality is determined mostly by decisions made before any model is involved. How documents are chunked — by fixed token count, or respecting headings and tables — determines whether a chunk contains a complete, answerable unit of information. How chunks are ranked — pure semantic similarity, or hybrid with lexical search — determines whether an exact term match beats a vaguely-related passage. Whether tables and structured data are extracted separately or flattened into prose determines whether numeric answers are even present in retrievable form.
The only reliable way to know if any of this is working is an evaluation set: a list of real questions with known-correct answers, checked against retrieval output directly, before the model ever generates a response. Without one, every reported improvement is a guess.
We build the eval set before we tune anything. It's the only way to tell whether the second version is actually better than the first, rather than just different.