The notebook version of retrieval-augmented generation takes an afternoon: load some documents, embed them, retrieve the top few chunks, ask the model. It answers the first ten questions well. Then someone asks about the policy that changed last quarter and the answer comes from the old version.
That failure is not unusual. A retrieval product has to choose the right source, the right passage and the right version before a language model writes anything. When an answer is wrong, teams often change the prompt or model. The more useful question is whether the system retrieved evidence that could support a correct answer.
Retrieval quality is the product
Chunking should follow the structure and use of the material. A contract clause, support procedure and technical table do not want the same chunk size. Preserve headings, document identity, version, owner and effective date as metadata. These fields let the system filter before searching and explain where an answer came from.
Use more than one retrieval signal. Keyword search handles exact product codes, legal phrases and names well. Semantic search handles paraphrases and concepts. Hybrid retrieval combines them. A reranker can then judge the smaller candidate set against the question. This costs time, so test whether it improves the questions that matter instead of adding it by default.
Freshness needs an operating process. Connectors must detect additions, updates and deletions. Failed ingestion needs an alert and an owner. Superseded material should be removed from active retrieval while remaining available where records rules require it. Access controls must travel from the source into the index so retrieval never becomes a side door around permissions.
Build the evaluation set before the pipeline
Collect real questions from the people who will use the system. Include simple lookups, questions requiring several passages, ambiguous requests and questions the material cannot answer. For each one, identify acceptable evidence and what a good answer must contain. Have domain reviewers grade results consistently.
- Retrieval checks whether the required evidence appears in the candidate passages.
- Answer checks whether the response is correct, complete and supported by those passages.
- Unsupported-claim checks flag statements that have no cited evidence.
- Refusal checks confirm that the system declines when evidence is absent or access is not allowed.
Run this set whenever chunking, models, prompts or source handling changes. Production feedback should add difficult questions to it. Without a stable evaluation set, each improvement is judged by whichever examples happen to be open in a browser.
Citations and refusals are features
A citation should lead to the exact passage and identify the source version. A list of document titles at the bottom is not enough. Users should be able to inspect the evidence where a claim appears. This makes review faster and gives the team diagnostic information when an answer is challenged.
The system also needs permission to say it does not know. Define minimum evidence conditions and ask for clarification when a question can mean several things. Confidence is better derived from retrieval and evaluation signals than from asking the model how sure it feels.
Cost, latency and the boring parts
Cache repeated retrieval and stable answers where policy allows it. Route simple questions to smaller models. Stream responses only when it improves the experience. Monitor ingestion delay, retrieval misses, answer quality, refusal behavior, latency and cost per useful answer. Sample real sessions for review with sensitive data protected.
Our Generative AI Development service treats retrieval as a maintained product: source governance, evaluation, user experience and operations are designed together. That is what turns a good notebook into a system people can rely on.
