Definition. Retrieval-Augmented Generation (RAG) is a pattern where relevant evidence is retrieved from an external knowledge source and supplied to a model as context before generation, so the model's answer is grounded in that evidence rather than relying solely on what it memorized during training.

Problem it solves. LLMs have a fixed knowledge cutoff (Chapter 4) and cannot know private, proprietary, or very recent information unless it is explicitly provided. Fine-tuning a model on new facts is slow, expensive, and does not reliably update discrete facts (Chapter 14). RAG solves this by keeping knowledge external and swappable — updating a document store is far faster and cheaper than retraining a model.

7.2 The Complete RAG Lifecycle

Sources -> parse / OCR if necessary -> clean & normalize -> chunk -> metadata + permissions -> embeddings / lexical index -> storage Question -> query understanding / rewrite -> retrieval -> hybrid fusion -> reranking -> context assembly -> generation -> citations -> evaluation / feedback

Fig 7.1 — The two halves of RAG: an offline/async ingestion pipeline (top) and an online query pipeline (bottom). Most production bugs originate in ingestion, not generation.

7.3 Ingestion

⚠️ Common Mistake Forgetting deletion propagation: when a source document is deleted or a user's access is revoked, the corresponding vectors/chunks must also be deleted or filtered out — otherwise retrieval can surface content that should no longer be accessible. This is both a correctness bug and a security defect (Chapter 12).

7.4 Chunking

Prerequisite Concept — Why Chunking Is Necessary at All

Intuition. Embedding models and context windows both have practical size limits, and retrieval quality degrades when a single "unit" of retrievable text mixes multiple unrelated topics (a 50-page document embedded as one vector produces a blurry average that matches nothing well). Chunking splits documents into smaller, more topically coherent units so each one can be embedded and retrieved precisely.

Strategy	How It Works	Trade-off
Fixed-size	Split every N tokens/characters	Simple, but can cut sentences/ideas mid-thought
Recursive/structure-aware	Split along headings, paragraphs, then fall back to smaller units	Respects document structure; more implementation effort
Sentence/paragraph	Split at natural language boundaries	Coherent units; sizes can vary widely
Semantic chunking	Split where embedding similarity between adjacent sentences drops	Topically clean chunks; more compute at ingestion time
Parent-child (small-to-big)	Retrieve small precise chunks, but expand to the larger parent section for generation context	Best of precision + context, more storage/complexity

Overlap trade-off. Adding overlap between consecutive chunks (e.g. the last 50 tokens of chunk N repeated as the first 50 tokens of chunk N+1) reduces the chance that a relevant sentence is split across a chunk boundary and therefore missed — but increases storage, duplicate content in retrieved results, and cost.

📌 Sticky Note — Remember Always preserve the source offset (page number, character range, section heading) for every chunk. Without it, a citation in the final answer cannot point back to verifiable evidence — and citation is one of RAG's core value propositions over an ungrounded chatbot.

7.5 Retrieval

Prerequisite Concept — BM25 (Sparse Retrieval)

Definition. BM25 is a classic keyword-ranking algorithm that scores documents based on term frequency (how often query terms appear), inverse document frequency (how rare/informative those terms are across the whole collection), and document length normalization. It is the modern refinement of TF-IDF and remains extremely strong for exact-term and rare-term matching.

Hybrid Search and Score Fusion

Definition. Hybrid search runs both dense (embedding) and sparse (BM25) retrieval in parallel, then combines their ranked result lists into one — commonly using Reciprocal Rank Fusion (RRF), which rewards documents that rank highly in either list without requiring the two very different scoring scales to be normalized against each other.

Query: "reset MFA on account 44821" Dense (semantic) results: Sparse (BM25) results: 1. "how to change 2FA settings" 1. "account 44821 setup guide" 2. "account security options" 2. "MFA reset procedure" 3. "login troubleshooting" 3. "two-factor authentication" │ │ └──────────── RRF fusion ──┘ │ ▼ 1. "MFA reset procedure" (ranked high in both) 2. "account 44821 setup guide" (exact match on ID) 3. "how to change 2FA settings"

Fig 7.2 — Hybrid fusion recovers the exact account-ID match that pure semantic search might rank lower, while still surfacing the semantically relevant "MFA reset" content.

Query Rewriting, Multi-Query, and Decomposition

Query rewriting/expansion. The user's raw query is transformed (by rules or a model call) into a clearer or more retrievable form — e.g. resolving pronouns using conversation history, or expanding an acronym. Multi-query retrieval generates several rephrased versions of the same question and retrieves for each, then merges results, improving recall against vocabulary mismatch. Query decomposition breaks a complex multi-hop question ("Which vendor had the highest late-delivery rate in Q2, and what's their contract renewal date?") into sub-questions that are retrieved and answered separately, then combined.

Reranking

Definition. A reranker is a second-stage model — typically a cross-encoder that jointly scores a (query, candidate) pair rather than comparing independent embeddings — that reorders an initial set of retrieved candidates by more precise relevance. Rerankers are more computationally expensive per item than embedding similarity, so they're applied only to the top-N candidates from the first-stage retrieval, not the whole collection.

7.6 Generation and Grounding

🚧 A Common Failure Teams often tune prompts when the real problem is retrieval. Always inspect retrieved chunks before blaming the model. If the correct evidence was never retrieved, no amount of prompt engineering on the generation step can fix the answer.

7.7 RAG Evaluation

RAG failures can occur at multiple independent layers, so evaluation must measure each layer separately rather than only judging the final answer.

Prerequisite Concept — Recall@k, Precision@k, MRR, NDCG

Layer	Questions to Measure	Example Metrics
Retrieval	Did we find the necessary evidence?	Recall@k, precision@k, MRR/NDCG, human relevance
Reranking	Did the best evidence move upward?	Ranking metrics, top-k relevance
Generation	Is the answer correct and supported?	Correctness, faithfulness, citation accuracy
End-to-end	Did the user get a useful answer?	Task success, human rating, latency, cost

Recall@k: of all the documents that are actually relevant to a query, what fraction appear somewhere in the top k retrieved results? Precision@k: of the top k retrieved results, what fraction are actually relevant? MRR (Mean Reciprocal Rank): the average of 1/rank of the first relevant result across many queries — rewards putting the best result near the top. NDCG (Normalized Discounted Cumulative Gain): a ranking-quality metric that rewards relevant results appearing earlier and can account for graded relevance (not just relevant/irrelevant).

Faithfulness vs Correctness

Comparison. Correctness asks whether the answer matches ground truth in the real world. Faithfulness asks whether the answer is actually supported by the retrieved evidence supplied to the model — an answer can be factually correct by coincidence while being unfaithful (not actually derived from the given context), which is a red flag that the system is not reliably grounded and may hallucinate on a query where its "guess" happens to be wrong.

7.8 Production RAG Project Checklist

Under the Hood — Multi-Tenant Isolation

Why it matters. In a multi-tenant RAG product (one deployment serving many customer organizations), a retrieval bug that omits the tenant filter can leak Customer A's confidential documents into Customer B's chat answers — one of the most damaging classes of AI application bugs. The filter must be enforced at the query layer (Chapter 6), tested explicitly (Chapter 12: cross-tenant retrieval tests), and re-verified whenever the retrieval code changes.

Common Questions

Q: Why use RAG instead of just fine-tuning the model on company documents?

RAG keeps knowledge external, so updating information is as fast as updating a document store, permissions can be enforced at retrieval time, and answers can be cited back to a verifiable source — fine-tuning bakes facts into weights, is slow to update, doesn't reliably teach discrete new facts, and provides no built-in citation mechanism.

Q: What happens if retrieval returns zero relevant chunks?

The system should explicitly tell the model (and the user) that no supporting evidence was found, rather than letting the model guess from unaided knowledge — this is a core grounding instruction, not an edge case to ignore.

Q: How would you debug a RAG answer that's wrong?

Inspect the retrieved chunks first — if the correct evidence wasn't retrieved, the bug is in ingestion/chunking/retrieval, not generation; only tune the generation prompt once you've confirmed the right evidence was actually supplied.

Q: What is the biggest limitation of RAG?

Answer quality is bounded by retrieval quality — a perfect generation prompt cannot compensate for evidence that was never found, poorly chunked, or filtered out incorrectly.

Interview / Viva Questions

Q: Explain the difference between dense and sparse retrieval, and why hybrid search combines them.

Dense retrieval uses embeddings to match semantic meaning even without shared words; sparse (BM25) retrieval matches exact/rare terms precisely. They have complementary failure modes, so hybrid search with fusion (e.g. RRF) captures the strengths of both.

Q: Why is reranking applied only to a shortlist rather than the whole collection?

Cross-encoder rerankers score each (query, document) pair jointly, which is far more computationally expensive than embedding similarity; applying it to the full collection would be prohibitively slow, so it's reserved for refining the top candidates from a cheaper first-stage search.

Q: What is the difference between faithfulness and correctness in RAG evaluation?

Correctness measures whether the answer matches real-world ground truth; faithfulness measures whether the answer is actually supported by the retrieved evidence given to the model — an unfaithful-but-correct answer is a warning sign of unreliable grounding.

Q: How do you prevent cross-tenant data leakage in a shared RAG deployment?

Enforce tenant/permission filters at the retrieval query itself (not post-filtering after generation), attach tenant IDs to every chunk at ingestion, and include explicit cross-tenant retrieval tests in the eval/security suite.

Q: What's the risk of setting chunk overlap too high or too low?

Too low: relevant content near chunk boundaries can be split and missed by retrieval. Too high: increased storage, duplicate/near-duplicate context passed to generation, wasted tokens and cost.

Scenario-Based Question

Q: Your RAG chatbot gives a plausible-sounding but wrong answer to a legal question, and the retrieved chunks show no relevant clause was found. What do you change?

First, tighten the generation instruction so the model explicitly declines to answer when evidence is insufficient rather than filling the gap from unaided knowledge. Second, investigate why retrieval missed the relevant clause — check chunking (was it split awkwardly?), indexing (was the document actually ingested?), and query understanding (did the question need rewriting/decomposition?). Add this case to the eval regression set so a future change doesn't reintroduce the same failure.

Chapter Summary

RAG grounds model output in externally retrieved evidence, avoiding the cost and staleness of fine-tuning for knowledge updates.
The pipeline has two halves: async ingestion (parse → chunk → embed → store) and online query handling (retrieve → fuse → rerank → generate → cite).
Chunking strategy and overlap directly determine retrieval recall and citation precision.
Hybrid search (dense + sparse, fused via RRF) outperforms either method alone.
Rerankers refine a cheap first-stage shortlist rather than scoring the entire collection.
Evaluation must separate retrieval, reranking, generation, and end-to-end quality — most real failures originate in retrieval, not generation.
Multi-tenant isolation and deletion propagation are correctness and security requirements.

7. Phase 5 — Retrieval-Augmented Generation (RAG)

7.1 Definition and Why RAG Exists