Retrieval-Augmented Generation

In Chapter 28 the model gained the ability to call tools. One of the most valuable things to fetch is KNOWLEDGE — facts, documents, and data the model was never trained on or cannot reliably recall. Retrieval-Augmented Generation (RAG) is the technique for grounding a model's answers in an external knowledge source: retrieve the relevant text, put it in the prompt, and let the model answer based on it. RAG is one of the most widely-used LLM techniques in production, and this chapter builds it from the ground up.

RAG mirrors how a careful person answers a hard question: rather than answering from memory alone, they LOOK IT UP in a reliable source, then answer based on what they found. RAG gives a model the same workflow. Given a question, the system first RETRIEVES the most relevant pieces of text from a knowledge base, inserts them into the prompt, and asks the model to answer USING that retrieved context. The model's broad language ability is combined with specific, up-to-date, trustworthy information.

RAG has two phases. An OFFLINE phase prepares the knowledge base (done once, ahead of time), and an ONLINE phase answers each query (done per question). Understanding the two phases and how they fit together is the foundation for everything else.

When a question arrives, the system retrieves relevant chunks and generates a grounded answer. This is the retrieve-then-generate flow — the heart of RAG:

The heart of modern retrieval is the EMBEDDING — a vector that captures the MEANING of a piece of text, building directly on the embeddings of Chapter 8. Dense retrieval uses embeddings to find text by SEMANTIC similarity: passages that MEAN the same thing as the query, even if they share no words. This is what lets RAG find 'parental leave' content when you ask about 'maternity time off'.

An embedding model converts any piece of text into a fixed-length vector — a list of numbers — positioned so that texts with SIMILAR MEANING have NEARBY vectors. 'How do I reset my password?' and 'I forgot my login credentials' land close together in the vector space, despite sharing almost no words, because they mean nearly the same thing. This semantic matching is the superpower of dense retrieval over keyword search.

To find the chunks most similar to the query, we compare their vectors. The standard measure is COSINE SIMILARITY — the cosine of the angle between two vectors (Chapter 1) — which is high when vectors point in the same direction (similar meaning) and low when they don't. Retrieval finds the k chunks whose vectors are most similar to the query vector.

Computing cosine similarity against every chunk works for a few thousand chunks, but real knowledge bases have millions or billions. Comparing the query to every single vector (a 'brute-force' search) becomes too slow. VECTOR DATABASES solve this with specialized indexes that find the nearest vectors quickly — the infrastructure that makes RAG scale.

Brute-force search compares the query against all N chunk vectors — O(N) per query. For millions of vectors and many queries per second, this is prohibitively slow. We need a way to find the nearest vectors WITHOUT comparing against all of them. The trick is to accept APPROXIMATE answers: find vectors that are ALMOST certainly among the nearest, far faster than guaranteeing the exact nearest.

ANN indexes organize vectors so that search can skip most of them. Two popular families: (1) HNSW (Hierarchical Navigable Small World) builds a navigable graph where search hops toward the query through a few well-connected nodes; (2) IVF (Inverted File) clusters vectors and searches only the nearest clusters. Both turn an O(N) scan into something far faster, at the cost of occasionally missing a true nearest neighbor.

FAISS (Facebook AI Similarity Search) is a widely-used library implementing these indexes. It is a LIBRARY (you embed it in your application), as opposed to managed vector DATABASES (Pinecone, Weaviate, Qdrant, Milvus, pgvector) that add storage, metadata filtering, and operations. For learning and many applications, FAISS is the workhorse; for production with persistence and scaling needs, a managed vector database often fits better.

Before documents can be embedded and indexed, they must be split into CHUNKS — smaller passages. Chunking is deceptively important: get it wrong and even great embeddings and search produce poor results, because the units being retrieved are the wrong size or split mid-thought. Beginners often overlook chunking; experts know it is one of the biggest levers on RAG quality.

Two reasons. First, embeddings work best on focused passages — embedding an entire 50-page document into one vector blurs its many topics into mush, so retrieval can't distinguish them. Second, the model's context window is finite — you can only fit so much retrieved text in the prompt, so you want to retrieve focused, relevant passages, not whole documents. Chunking creates retrievable units that are focused enough to embed well and small enough to fit in context.

The art is finding a chunk size that is focused enough to embed and retrieve precisely, yet large enough to contain a complete idea. There is no universal answer — it depends on your documents and queries — but a few hundred tokens with some overlap is a common starting point.

Dense (semantic) retrieval and sparse (keyword) retrieval each have strengths and weaknesses. HYBRID SEARCH combines them to get the best of both — and in practice, hybrid search usually beats either alone. Understanding why requires seeing exactly where each method shines and fails.

The pattern is clear: dense retrieval excels at MEANING (synonyms, concepts, fuzzy questions), while keyword search excels at EXACTNESS (specific terms, names, codes, rare words). A query like 'error code TX-409 in the billing module' needs BOTH — the exact code (keyword) and the conceptual context of billing errors (dense). Hybrid search runs both and combines the results.

The standard way to merge dense and keyword results is RECIPROCAL RANK FUSION (RRF): each method ranks the chunks, and a chunk's combined score is based on its RANK in each list (not its raw score, which aren't comparable across methods). Chunks that rank highly in EITHER method bubble to the top. This is simple, robust, and avoids the problem of dense and keyword scores being on different scales.

Initial retrieval (dense, keyword, or hybrid) is fast but coarse — it casts a wide net to find candidate chunks. RERANKING adds a second, more precise pass: take the top candidates from retrieval and re-score them with a more powerful (but slower) model that judges relevance more accurately. This two-stage 'retrieve-then-rerank' approach significantly improves the quality of the final context.

Fast retrieval embeds the query and each chunk SEPARATELY, then compares vectors — the chunk's vector was computed without ever seeing the query. A RERANKER instead looks at the query and a candidate chunk TOGETHER, in one model pass, judging how well that specific chunk answers that specific query. This 'cross-encoder' approach is far more accurate, but too slow to run over millions of chunks — so it is applied only to the top candidates from fast retrieval.

Once the best chunks are retrieved and reranked, they must be assembled into the prompt — 'context stuffing'. This step seems trivial but has real subtleties: how much to include, in what order, and how to instruct the model to use it. Done poorly, even perfect retrieval yields bad answers.

The prompt instructions matter as much as the retrieved text. Three instructions are crucial: (1) 'answer using ONLY the sources' — grounds the model in the retrieved text; (2) 'if the answer isn't in the sources, say you don't know' — prevents the model from filling gaps with hallucination; (3) 'cite sources by number' — makes answers verifiable and traceable. These instructions are what turn retrieved text into trustworthy, grounded answers.

A surprising and important finding (Liu et al., 2023): models pay MORE attention to information at the BEGINNING and END of the context than in the MIDDLE. If you stuff ten chunks and the crucial one lands in the middle, the model may overlook it — a phenomenon called 'lost in the middle'. This is why reranking matters beyond just filtering: placing the most relevant chunks at the start (and end) of the context, not buried in the middle, improves the model's use of them.

RAG systems fail in characteristic ways, and — as stressed in Section 29.2 — most failures are in the RETRIEVAL half, not the generation. Knowing the failure modes and their fixes makes debugging systematic.

When a RAG answer is wrong, the systematic first step is to INSPECT WHAT WAS RETRIEVED. Print the chunks that went into the prompt. Usually one of two things is true: either the right chunk was NOT retrieved (a retrieval problem — fix chunking, embeddings, k, or add hybrid/reranking), or the right chunk WAS retrieved but the model didn't use it (a generation problem — fix the prompt instructions or chunk ordering). Distinguishing these two cases tells you exactly where to focus.

The basic retrieve-then-generate pipeline can be extended in many ways to handle harder cases. These advanced patterns address specific weaknesses of vanilla RAG and are increasingly common in production systems.

Users phrase questions in ways that don't match how documents are written. Query rewriting uses the model to rephrase or expand the query before retrieval — turning a terse 'parental leave?' into 'What is the company parental leave policy, including duration and eligibility?' — which retrieves better. Multi-query retrieval generates several phrasings and merges their results, improving recall.

The most important modern pattern connects directly to Chapter 28: treat retrieval as a TOOL the model can call. Instead of always retrieving once at the start, the model DECIDES when it needs to look something up, formulates the query itself, and can retrieve multiple times as it works through a problem (the ReAct loop). This 'agentic RAG' handles complex, multi-step questions that a single retrieval cannot — the model retrieves, reasons, retrieves again, and synthesizes.

RAG is one of three ways to get a model to use specific knowledge. The others are FINE-TUNING (bake the knowledge into the weights, Chapter 22) and LONG CONTEXT (just put everything in the prompt). Each has its place, and knowing when to use which is an important practical judgment.

For injecting KNOWLEDGE (facts, documents, data), RAG has decisive advantages over fine-tuning: it updates instantly (just change the index — no retraining), scales to far more knowledge than fits in weights or context, provides citations, and keeps knowledge separate from the model so it's auditable and current. Fine-tuning is better for teaching SKILLS, STYLE, or FORMAT — how to behave — rather than facts to recall.

RAG and fine-tuning are not mutually exclusive — the strongest systems often use both: fine-tune the model for the domain's style and the skill of using retrieved context well, AND use RAG to supply current, specific knowledge at query time. And as context windows grow (Chapter 33), the line between RAG and long context shifts — but even with huge contexts, retrieval remains valuable for selecting WHAT to put in the context from a knowledge base far too large to fit entirely.

Let us assemble the whole chapter into a complete, production-minded RAG system, integrating chunking, embedding, indexing, hybrid retrieval, reranking, context assembly, and grounded generation.

Exercises 1–10 are pen-and-paper or derivations; 11–22 require code.

Further reading: “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (Lewis et al., 2020) — the original RAG paper. “Dense Passage Retrieval” (Karpukhin et al., 2020). “Billing-scale similarity search with GPUs” and the FAISS library (Johnson et al., 2017). “HNSW” (Malkov & Yashunin, 2018) for graph-based ANN. “Lost in the Middle” (Liu et al., 2023). “Precise Zero-Shot Dense Retrieval (HyDE)” (Gao et al., 2022). “Self-RAG” (Asai et al., 2023). “Contextual Retrieval” (Anthropic, 2024). The RAGAS framework for RAG evaluation.

Next → Chapter 30: Multi-modal LLMs

So far our models work entirely in text. But the world is not only text — it is images, audio, and video. Chapter 30 extends LLMs to MULTIPLE MODALITIES: models that can SEE images and HEAR audio, not just read text. We will see how images are turned into tokens the model can process (vision encoders and projection), how text and visual information are fused, how these models are trained, and what they can do — from describing images to answering questions about charts and documents. The text-only assistant becomes one that perceives the world more like we do.