Course module

RAG Fundamentals

RAG is not only retrieval plus generation. A useful RAG system is a trust pipeline that decides what knowledge is allowed, what evidence is relevant, how fresh it is, whether the user can see it, and how the answer should admit uncertainty.

How this module was made: I study public references, then rewrite the concept in my own words with a scenario, mental model, checklist, diagram, and practice prompts. The goal is to make the idea easier to understand here, not send you away to read raw markdown.

Why this matters

RAG is usually the first production architecture teams reach for, but naive retrieve-then-generate systems often fail because of poor chunks, weak ranking, missing metadata, stale sources, permission leakage, and no way to debug answer quality.

Concrete scenario

Scenario: HR policy assistant for employees

Imagine an internal assistant that answers HR policy questions. The documents include old PDFs, updated handbook pages, leave policy announcements, location-specific rules, and manager-only notes. Some employees are in India, some are in the US, and some documents should not be visible to everyone.

User question

Can I carry forward unused leaves into next year if I joined in October?

The answer depends on the employee location, joining date, policy version, document freshness, and permissions. A generic semantic search result can easily retrieve an outdated policy or a broad handbook page that does not answer the exact case.

Visual mental model

Use this diagram as the quick recall map before reading the detailed points.

RAG Fundamentals diagram

Rendering diagram...

Bad v1 design

This is the common demo-first version that looks fine until real users ask messy questions.

Upload every PDF into a vector database without document ownership or freshness metadata.
Split text into fixed-size chunks even when tables, policy sections, and headings need to stay together.
Search only by embedding similarity and pass the top five chunks directly to the model.
Ask the model to answer even when the retrieved evidence is weak, old, or not specific to the user context.
Show citations only at document level, making it hard for the user to verify the exact claim.

Better v1 design

This version is still simple, but it is easier to trust, debug, and improve.

Start with trusted sources and assign owners, update dates, regions, access rules, and document types before ingestion.
Use structure-aware chunking so headings, tables, policy clauses, and exceptions remain understandable.
Classify the user question first: factual lookup, policy comparison, procedure, exception case, or unsupported request.
Retrieve with metadata filters, hybrid search, and reranking; then build a compact context pack with source snippets and freshness signals.
Generate an answer with citations, uncertainty language, and a fallback path when evidence is missing or conflicting.

Key ideas

Fine-tuning is better for repeated behavior, tone, and format; RAG is better for changing facts, citations, and source-grounded answers.
The retrieval layer is not only vector search. It usually needs query rewriting, filters, hybrid search, reranking, deduplication, and context assembly.
Metadata is as important as embeddings. Source owner, date, region, document type, access rules, and version decide whether retrieved text can be trusted.
Long context does not remove the need for retrieval. It only changes how much filtered evidence can be passed into the model.
A production RAG system should be designed as a trust system, not as a vector database wrapper.

Design checklist

Define trusted sources and ownership before ingestion.
Capture metadata: source, version, owner, access level, region, document type, and last updated date.
Choose chunking based on the questions users ask, not only on token limits.
Use hybrid search and reranking when semantic search alone retrieves broad or stale chunks.
Return citations, freshness signals, and confidence boundaries with answers.
Evaluate retrieval quality separately from answer quality so failures are easier to debug.

Evaluation signals

Use these checks to know whether the system is actually improving.

Signal	Question to ask	Example failure
Retrieval precision	Did the system retrieve the exact policy section needed for this question?	Broad handbook section retrieved instead of the carry-forward rule.
Freshness	Is the retrieved source current enough for the user decision?	Old PDF retrieved because it had similar wording.
Permission safety	Was every retrieved source visible to this user?	Manager-only clarification leaked into employee answer.
Groundedness	Can every claim be traced to a source snippet?	Model added a condition that was not in the policy.
Usefulness	Did the answer help the user decide what to do next?	Answer was technically correct but did not mention who to contact or what form to fill.

Interview prompts to practice

When would you use RAG instead of fine-tuning?

Design production RAG for a company knowledge base with document permissions.

How would you debug a RAG system that gives fluent but unsupported answers?

How would you handle stale documents and conflicting sources?

Practice exercise

Use this to turn the note into an actual design exercise.

Pick one domain: HR policies, college notices, pharmacy FAQs, or product documentation.
List five document types and the metadata each one needs.
Write three user questions: one simple lookup, one exception case, and one unsupported request.
Design the retrieval flow from question classification to citations.
Define five test cases that would catch retrieval, freshness, permission, and hallucination failures.

How to study this module

Read the scenario first and explain why the design is hard.
Use the diagram to explain the flow out loud in two minutes.
Compare bad v1 and better v1 before jumping to tools.
Practice the interview prompts using the checklist.
Apply the idea to one project, diagram, or architecture review.

Continue course Read blogs