Deep-dive blog
How I Would Build a Production RAG Copilot
I used to think RAG starts with embeddings. Now I think that is the wrong starting point. A useful RAG copilot starts with a person, a question, a decision, and the evidence needed to trust the answer.
Many teams build their first RAG demo by embedding documents, retrieving a few chunks, and asking a model to answer. That can impress people in a demo. It does not survive real usage for long.
Real users ask incomplete questions. Documents go stale. Policies have exceptions. Some sources are restricted. Tables lose meaning when chunked badly. A model can produce a fluent answer even when the retrieved context is weak. That is why I do not think of RAG as retrieval plus generation. I think of it as a trust pipeline.
In this article, I will use a concrete HR policy assistant example and walk through the design I would actually start with: bad v1, better v1, architecture, evaluation, failure modes, and what I would ship first.
Scenario
An HR assistant answering leave policy questions
Imagine an employee asks: Can I carry forward unused leaves into next year if I joined in October?
This sounds simple, but the answer may depend on region, joining date, employment type, policy year, manager approval, and whether the source is the current handbook or an old PDF. If the system retrieves a stale policy with similar wording, the final answer can be fluent and still wrong.
The bad v1
This is the common version that looks good in a demo and becomes painful in production.
- •Ingest every PDF and help-center page into a vector database with no owner, date, region, or access metadata.
- •Split every document into equal token chunks, even when headings, tables, policy exceptions, and examples should stay together.
- •Retrieve the top five semantically similar chunks and pass them directly into the model.
- •Ask the model to answer every question even when evidence is old, conflicting, or not specific enough.
- •Show a broad document citation instead of the exact source snippet that supports each claim.
The better v1
This is still small, but it is easier to trust, debug, and improve.
- •Start with one narrow workflow, such as HR leave policy questions, instead of trying to answer every company question.
- •Ingest only trusted sources and attach metadata: source owner, update date, region, access group, document type, and version.
- •Classify the question before retrieval: factual lookup, exception case, procedure, comparison, or unsupported request.
- •Use metadata filters, hybrid search, reranking, and deduplication before building the final context pack.
- •Return an answer with citations, freshness signals, missing-evidence language, and a feedback path.
Reference architecture
The important part is not the vector store. The important part is the evidence path from user context to final answer.
How I would build it over three weeks
Week 1: make one workflow trustworthy
- •Pick one workflow: HR leave policy, product docs, pharmacy FAQs, or internal SOPs.
- •Collect 20 to 50 high-quality questions with expected source documents.
- •Ingest only trusted documents and capture metadata before embeddings.
- •Make citations mandatory from the first version.
Week 2: separate retrieval quality from answer quality
- •Create a retrieval eval set where the expected source section is known.
- •Track whether the correct chunk appears in top 1, top 3, and top 5.
- •Add reranking only after measuring baseline retrieval.
- •Log query rewrite, filters, retrieved chunks, reranked chunks, and final evidence pack.
Week 3: add answer review and feedback loop
- •Check whether every answer claim is grounded in the evidence pack.
- •Add user feedback categories: wrong source, stale source, unclear answer, missing detail, useful answer.
- •Create a review queue for high-risk or low-confidence answers.
- •Use failures to improve data quality, retrieval filters, and prompt instructions.
Evaluation table
I would not call the system production-ready until these signals are visible.
| Signal | Test | Why it matters |
|---|---|---|
| Retrieval precision | For 50 known questions, did the correct source section appear in the top three results? | If the right evidence never reaches the model, prompt tuning will not save the answer. |
| Citation coverage | Can every important claim in the answer point to a specific source snippet? | A citation to a broad document is not enough when users need to trust a policy answer. |
| Freshness | Does the system prefer current policy pages over older PDFs with similar wording? | Stale retrieval is one of the easiest ways to give confidently wrong answers. |
| Permission safety | Can the system prove no restricted document was used for a user without access? | RAG systems can leak information through retrieved context even if the final answer looks harmless. |
| Abstention quality | When evidence is missing or conflicting, does the system ask for clarification or route to a human? | A useful assistant should know when not to answer. |
Mistakes I would avoid
- •Starting with the vector database instead of the user workflow.
- •Treating chunking as a token-size problem instead of a meaning-preservation problem.
- •Adding more documents before you can debug the first small corpus.
- •Evaluating only final answers and ignoring retrieval quality.
- •Trusting fluent answers without source-level verification.
What makes this production work
The system becomes useful when every answer can be traced back to evidence, every bad answer can be debugged, and every improvement can be linked to a data, retrieval, prompt, or product change.
That is the part beginners miss. RAG quality is not only about getting better embeddings. It is about designing the complete path from user context to trusted answer.
The practical takeaway
A production RAG copilot should be evaluated like a product system: did the right evidence reach the model, did the answer stay grounded, did the user receive useful citations, and can the team debug failures when the answer is wrong?