RAG Fundamentals

RAG is not a vector database plus a prompt.

A useful RAG system is a trust pipeline. It decides which knowledge is allowed, which source is current, which evidence is relevant, whether the user can see it, and how the final answer should explain uncertainty.

If you remember only one line from this lesson, remember this:

RAG quality is not only answer quality. It is evidence quality plus answer quality.

Scenario: HR policy assistant

Imagine we are building an internal assistant for employees.

A user asks:

Can I carry forward unused leaves into next year if I joined in October?

This sounds simple. It is not.

The answer may depend on:

• employee location
• joining date
• employment type
• current policy year
• old policy PDFs
• newer handbook pages
• manager-only clarifications
• leave category
• exceptions for probation or notice period

A demo system may retrieve a paragraph that looks similar and answer confidently. A production system must know whether the evidence is specific, current, allowed, and enough.

Why naive RAG fails

The naive version looks like this:

This is useful for a demo. It is dangerous as the whole design.

What goes wrong

Bad v1 design

A bad v1 usually has these decisions:

1. Ingest everything quickly.
2. Split by fixed token size.
3. Store only text and embedding.
4. Retrieve top five chunks.
5. Ask the model to answer.
6. Add citations later.
7. Evaluate by reading a few outputs manually.

This is exactly how many demos start. The issue is not that it is useless. The issue is that it hides the real failure points.

Better v1 design

A better v1 is still simple, but it is designed for debugging.

The important change is not just adding more components. The important change is that every step creates a signal we can inspect.

Step 1: define trusted sources

Before embeddings, decide what sources are allowed.

For the HR assistant, a source inventory may look like this:

This table matters because retrieval quality starts before retrieval. If sources are messy, answers will be messy.

Step 2: capture useful metadata

Embeddings do not know whether a document is current, restricted, official, or location-specific.

Add metadata like:

Metadata lets the system answer questions like:

• Is this source current?
• Does this user have access?
• Is this region-specific?
• Is this a policy, FAQ, or informal note?
• Should this source override an older document?

Step 3: classify the question

Not every question should be handled the same way.

The question:

Can I carry forward unused leaves into next year if I joined in October?

is not a simple keyword lookup. It is an exception case.

A useful classifier might label it as:

This classification decides retrieval filters and answer style.

Step 4: retrieve evidence, not just text

The retrieval target should be an evidence pack.

An evidence pack contains:

• source title
• exact snippet
• section heading
• effective date
• access level
• why it was selected
• whether it conflicts with another source

Example:

Now the model has something structured to reason from.

Step 5: answer with boundaries

A good answer should not just say yes or no.

It should say:

• what the policy says
• what assumptions were used
• where the answer came from
• what is uncertain
• what the user should do next

Example answer shape:

Notice the answer does not pretend to know everything. It gives a useful answer and marks the boundary.

Evaluation signals

Do not evaluate only final answers.

Evaluate the full path.

Debugging playbook

When an answer is wrong, do not immediately change the prompt.

Ask where the failure happened:

1. Source failure: Was the right document ingested?
2. Metadata failure: Was the source date, region, or access rule missing?
3. Retrieval failure: Did the right source fail to appear?
4. Ranking failure: Did the right source appear but too low?
5. Context failure: Was the evidence pack too noisy?
6. Generation failure: Did the model ignore or distort evidence?
7. UX failure: Did the answer omit next steps or uncertainty?

This is how a RAG system becomes improvable.

What I would ship first

For a first production version, I would not ingest the whole company knowledge base.

I would ship:

• one workflow
• one department
• 20 to 50 trusted documents
• 50 gold questions
• mandatory citations
• retrieval logs
• answer feedback
• review queue for low-confidence answers

That is enough to learn whether the architecture works.

Practice exercise

Pick one domain:

• college notices
• pharmacy FAQs
• HR policies
• product documentation
• internal SOPs

Then answer these:

1. What are the trusted sources?
2. Who owns each source?
3. What metadata is required?
4. What are three common user questions?
5. Which question is simple lookup?
6. Which question is an exception case?
7. What should the system refuse to answer?
8. What five eval cases will catch bad retrieval?

Summary

RAG is not just a retrieval pattern. It is a product-quality system.

A good RAG system does five things well:

1. Finds the right evidence.
2. Filters by user and source context.
3. Builds a compact evidence pack.
4. Answers with citations and uncertainty.
5. Produces debug signals when it fails.

If you design only for generation, you get demos.

If you design for evidence, evaluation, and debugging, you get a system people can trust.