Retrieval-Augmented Generation (RAG)

TL;DR

Intuition

A pretrained LLM is a compressed snapshot of its training corpus. Three problems follow from that fact [9].

Knowledge cutoff. Anything that happened after the training data freeze is invisible. The model cannot learn it without further training.

Hallucination under uncertainty. When the model is asked something it does not know, it cannot say “I do not know”; the next-token distribution is always defined. The model produces a plausible continuation, which often looks correct on the surface but is wrong on the facts.

No attribution. The model cannot point at a source for a claim. Even when its parametric memory is correct, you have no audit trail.

Retrieval addresses all three at once. Instead of trusting parametric memory, fetch the relevant passage from an external corpus and put it in the context window. Now the model has fresh data, has a citable source, and can be held to account when it strays from the retrieved evidence.

What retrieval does not fix: it does not improve reasoning, it does not make a 7B model behave like a 70B, and it does not help when the corpus itself is wrong or missing. RAG is a knowledge prosthesis, not a capability multiplier.

1. 2020Lewis et al.: RAG (NeurIPS)paper

   Joint training of a DPR-style retriever and a BART generator over Wikipedia. Introduces RAG-Sequence and RAG-Token marginalization. The first end-to-end RAG paper and the source of the acronym.

2. 2020REALM: retrieval inside pretrainingpaper

   Guu et al. wire a learned retriever into masked language modelling pretraining itself, so the LM is rewarded for fetching helpful evidence. Establishes that retrieval can be trained as part of the LM, not just bolted on at fine-tuning.

3. 2020FiD: Fusion-in-Decoderpaper

   Izacard and Grave encode each retrieved passage independently in the encoder and let the decoder cross-attend to the concatenated stack. Wins NaturalQuestions and TriviaQA at the time and becomes the default open-domain QA architecture.

4. 2022HyDE: hypothetical document embeddingspaper

   Gao et al. generate a hypothetical answer with the LLM, embed that, and retrieve against it. Closes a chunk of the zero-shot dense-retrieval gap without any labelled query-document data.

5. 2024GraphRAGsystem

   Microsoft Research generalize RAG to a knowledge graph extracted from the corpus by an LLM. Community-summary retrieval gives query-focused summarization that flat chunk retrieval cannot match on global questions.

Anatomy of a RAG system

The retrieve-then-read pipeline has six stages, each of which is a separate engineering decision.

1. Indexing. Documents are split into chunks, each chunk is embedded into a vector, and the vectors are stored in a search structure (an inverted index for sparse, an ANN index for dense). The chunk size, the splitter, and the embedding model are all chosen at this stage and are expensive to change later because the whole corpus must be re-indexed.
2. Query encoding. The user query is converted into the same representation as the chunks: a token sparse vector (BM25), a single dense vector (DPR / SBERT), or a multi-vector representation (ColBERT). The query encoder must match the document encoder for dense methods.
3. Retrieval. The system fetches the top-$k$ chunks by similarity. Sparse methods score with BM25; dense methods score by inner product or cosine; hybrid methods linearly combine the two.
4. Reranking (optional). A second-stage cross-encoder scores each (query, chunk) pair jointly and re-orders the top-$k$. This is the highest-quality but most expensive stage; it is typically run on $k = 100 \to k' = 10$.
5. Augmentation. The retrieved chunks are formatted into a prompt template along with the user query. The template names the chunks (often as [1], [2], etc.) and asks the model to cite them in the answer.
6. Generation. The LLM produces an answer conditioned on the context. Optionally a post-hoc verifier checks every claim against the retrieved chunks and edits or refuses unsupported claims.

Each stage has its own failure mode. Bad chunking loses information at chunk boundaries. A weak embedder misses semantic matches. A retriever with too small a $k$ misses the relevant chunk. A bad prompt template fails to ground the generator on the retrieved context. A generator that is too small ignores the context and falls back on parametric memory. The full system is only as good as its weakest stage.

Sparse retrieval: BM25 and the probabilistic relevance framework

Sparse retrieval treats a document as a bag of terms and scores it by the sum of per-term contributions. The dominant scoring function, BM25, is the empirical sweet spot of decades of probabilistic-IR research [8].

BM25 is still surprisingly competitive in 2024. Its strengths: zero training cost, transparent ranking, exact-term match (which handles names, acronyms, IDs, and rare technical terms that out-of-vocabulary embeddings stumble on), and trivially-updatable indexes. Its weaknesses: it cannot match a query and a document that share meaning but no surface terms (“car” versus “automobile”; “renal failure” versus “kidney disease”), it has no notion of word order beyond document level, and it does not transfer across languages.

Dense retrieval: DPR and the contrastive loss

Dense retrieval encodes the query and each candidate passage into a fixed-dimensional vector and scores by inner product. The encoders are trained so that vectors of relevant query-passage pairs are close and vectors of unrelated pairs are far apart [4].

After training, the passage encoder is run once over the whole corpus to produce a dense matrix $\mathbf{P} \in \mathbb{R}^{N \times d}$. At query time the query is encoded once and the inner product against all $N$ rows is approximated with an ANN index (FAISS HNSW, IVF-PQ, ScaNN). Latency is under $20$ms for ten million chunks on commodity hardware.

Embedding models: SBERT, E5, BGE, and the closed APIs

Sentence-BERT [5] was the first practical recipe for general-purpose sentence embeddings. The trick: take a pretrained BERT, attach a mean-pooling layer over the last-hidden-state tokens, and fine-tune with a Siamese architecture on natural-language-inference pairs (entailment / contradiction / neutral). The resulting embeddings make cosine similarity between sentences a reasonable proxy for semantic similarity, and they were the first to push BERT-derived embeddings past pre-deep-learning baselines on STS benchmarks at retrieval-friendly speeds.

Modern embedding models build on the SBERT scaffold but train at much larger scale and with explicit retrieval-oriented data:

• E5 (Microsoft). A T5-style encoder pretrained on a large web crawl as a contrastive denoising task, then fine-tuned on MS MARCO with hard negatives. Strong on MTEB at multiple sizes (small / base / large).
• BGE (BAAI). Beijing Academy of AI’s general embedding family, trained with a recipe similar to E5 but with more aggressive hard-negative mining and instruction-prefix prompts. The current open-weights leader on most retrieval benchmarks at the base size.
• GTE (Alibaba). A T5-encoder family with multilingual variants. Particularly strong at non-English retrieval.
• OpenAI text-embedding-3-small / large. Closed-API, but Matryoshka-trained: the same embedding can be truncated to lower dimensions without retraining, trading accuracy for index size.
• Cohere Embed v3. Closed-API, with a “search query” versus “search document” mode that prefixes the input differently and reduces the asymmetry between query and document representations.

For most production workloads the right default in 2024 is BGE-base or BGE-large for English, GTE-multilingual for non-English, and either the OpenAI or Cohere API when the team would rather pay per call than host the model.

Sparse, dense, hybrid: which retriever for which job

The standard hybrid recipe is Reciprocal Rank Fusion: run BM25 and the dense retriever independently, take the top $k$ from each, then rank by

with a small smoothing constant $c \approx 60$. RRF is parameter-light and beats most hand-tuned weighted-sum schemes. The mental model: each retriever votes for its top documents, and a document that appears near the top in both gets the most votes. Production systems on diverse query workloads consistently see hybrid beat either component on its own by 2-4 absolute points of recall@20.

Vector databases: where the embeddings live

The vector database is an ANN index plus a metadata store plus an API. The four common shapes:

• FAISS (Meta). A library, not a database. In-process, no server, no metadata store; you write the persistence layer yourself. The fastest option per query when the index fits in RAM. Used as the engine inside many higher-level systems.
• Pinecone (managed SaaS). Fully managed cloud, pay-per-vector. The fastest path to production for teams that do not want to run a service. Indexes are pod-based and you scale by pod type and replica count.
• Weaviate (open-source). Self-hostable graph-aware vector DB with native hybrid search and a GraphQL-style API. Good fit when you want to query both structured fields and vectors in one call.
• Milvus (open-source). Distributed vector DB designed for the billion-scale regime, with multiple ANN backends (HNSW, IVF, DiskANN). Strong on operational tooling for the largest indexes.
• pgvector (Postgres extension). A vector type and HNSW / IVFFlat indexes inside Postgres. The right choice when the team already runs Postgres and the embedding workload is moderate (under a hundred million vectors). Hybrid with full-text search and structured filters comes for free because everything is one SQL query.

For most teams in 2024 the right default is pgvector if they already have Postgres, otherwise managed Pinecone. Self-hosted Weaviate or Milvus is worth the operational burden only when scale, cost, or feature requirements actually demand it.

Chunking: where the corpus meets the index

Chunking is the boring stage that quietly wrecks more RAG systems than any other. The retriever can only return chunks; if the answer to a query straddles a chunk boundary, no retriever recovers it.

The four common strategies:

• Fixed-size token chunks. Split into $L$-token windows with $o$-token overlap (typical $L = 512$, $o = 64$). Cheap, language-agnostic, and a strong baseline. Loses information at hard boundaries inside paragraphs and sentences.
• Sentence-aware chunks. Split on sentence boundaries first, then group up to $L$ tokens. Avoids splitting mid-sentence; needs a tokenizer-aware sentence splitter.
• Recursive structural chunks. Walk the document tree (Markdown headings, HTML sections, code-file functions). Try the largest unit first; if it exceeds $L$, recurse into its children. Preserves document structure and is the LangChain RecursiveCharacterTextSplitter default.
• Semantic chunking. Embed every sentence, then place a chunk boundary wherever the cosine similarity between consecutive sentences drops below a threshold. The most expensive option; the gain over recursive structural chunking is small on most corpora.

Two heuristics that hold across corpora: always overlap chunks by 10-20% of the chunk size to soften boundary loss, and prepend each chunk with breadcrumbs (the document title and section heading) so the embedding does not depend on the chunk being interpretable in isolation.

ColBERT and late interaction

Single-vector dense retrieval compresses an entire passage into one $d$-dimensional vector. Most of the per-token information is averaged away. Late-interaction models like ColBERT keep the per-token vectors and defer the comparison to query time [6].

Advanced techniques

The retrieve-then-read baseline is the floor, not the ceiling. A handful of techniques routinely improve it.

HyDE (Hypothetical Document Embeddings). Ask the LLM to draft a hypothetical answer to the query, then embed and retrieve against that hypothetical answer instead of the raw query [7]. The hypothetical answer often contains the same vocabulary as the real answer and lands closer in embedding space than the bare query. Wins consistently on zero-shot dense retrieval where there is no labelled training data to fine-tune the encoder.

Query expansion. Have the LLM rewrite the query in several ways (synonym expansion, decomposition into sub-questions, formal-language paraphrase) and retrieve against each rewriting; merge the results. Costs an extra LLM call but typically lifts recall@10 by a few points, especially on conversational queries.

RAG-Fusion. Combine multi-query retrieval with RRF fusion. Generate $N$ rewritings, run each through the retriever, then RRF-fuse their top-$k$ lists. A small but reliable lift over single-query retrieval.

Reranking with cross-encoders. After top-$k$ retrieval, score each (query, chunk) pair jointly with a cross-encoder (such as bge-reranker-large or a Cohere rerank API). This is more expensive than dual-encoder retrieval but more accurate, and it is run on the top-$k$ only. Typical setup: dense retrieval $k = 100$, rerank to $k' = 10$, generate.

Agentic RAG. Let the LLM call the retriever as a tool, observe the results, decide whether they answer the question, and re-query if they do not. The agent can also decompose multi-hop questions into sub-questions and retrieve for each. More expensive (multiple retrieval rounds) but the right shape for hard multi-hop or vague queries.

GraphRAG. Extract entities and relations from the corpus with the LLM, build a knowledge graph, summarize each community of related entities, then retrieve community summaries instead of (or alongside) flat chunks. Wins on global / query-focused-summarization questions (“what are the main themes in this corpus?”) that flat-chunk retrieval cannot answer because no single chunk contains the global view [1].

Evaluation: RAGAS and the four metrics that matter

End-to-end RAG evaluation is harder than either retrieval evaluation or LM evaluation in isolation, because errors compound across stages. The RAGAS framework decomposes evaluation into four metrics, each scoring a different stage of the pipeline.

Faithfulness. The fraction of claims in the generated answer that are supported by the retrieved context. Measure by extracting atomic claims from the answer with an LLM, then for each claim asking the same LLM “is this claim entailed by the context?”. Low faithfulness signals hallucination: the generator is making things up despite having relevant context.

Answer relevance. Whether the answer addresses the question. Measure by having an LLM generate $N$ questions for which the answer is a plausible response, then compute the average cosine similarity between those generated questions and the original. Low relevance signals off-topic answers.

Context precision. Of the retrieved chunks, how many are actually relevant to the query. Measure with an LLM judge, position-weighted so that relevant chunks should appear at the top. Low precision signals a noisy retriever whose top-$k$ contains too many distractors.

Context recall. Of the chunks needed to answer the query, how many are in the retrieved top-$k$. Requires ground-truth answers (from a labelled QA dataset) and an LLM to check whether each ground-truth claim is supported by the retrieved context. Low recall signals a missing-fact problem: the retriever did not fetch what the generator needed.

The four metrics fail differently and the diagnosis is mechanical. High recall + high precision + low faithfulness: the generator is hallucinating despite good context, fix the prompt or change the model. High precision + low recall: the retriever is missing chunks, increase $k$ or rerank. Low precision + high recall: too much noise in the top-$k$, add a reranker. Low answer relevance with everything else healthy: the prompt template is not channeling the context into the answer, fix it. RAGAS is not the only framework (TruLens and DeepEval ship similar metrics), but the four-metric decomposition is what separates production RAG evaluation from “LLM-as-judge gives it a 7” eyeballing.

Wiring it up: LangChain, LlamaIndex, PyTorch + FAISS

Three production-shaped recipes for the same retrieve-then-read pipeline. LangChain has the largest ecosystem of pluggable components. LlamaIndex was built around indexing and retrieval first and has the cleanest abstractions for hybrid retrieval and metadata filtering. PyTorch + FAISS is the no-framework path: more code but full control and no upgrade risk.

from langchain_community.document_loaders import DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import FAISS
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

# 1. Index.
docs = DirectoryLoader("corpus/", glob="**/*.md").load()
splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=64)
chunks = splitter.split_documents(docs)
embed = HuggingFaceEmbeddings(model_name="BAAI/bge-base-en-v1.5")
store = FAISS.from_documents(chunks, embed)

# 2. Retrieve + read.
retriever = store.as_retriever(search_kwargs={"k": 6})
prompt = PromptTemplate.from_template(
  "Answer the question using ONLY the context.\n"
  "Cite chunks like [1], [2].\n"
  "If unsupported, say 'I do not know'.\n\n"
  "Context:\n{context}\n\nQuestion: {question}"
)
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa = RetrievalQA.from_chain_type(
  llm=llm,
  retriever=retriever,
  chain_type="stuff",
  chain_type_kwargs={"prompt": prompt},
  return_source_documents=True,
)
result = qa.invoke({"query": "How does BM25 saturate term frequency?"})
print(result["result"])
for d in result["source_documents"]:
  print("-", d.metadata.get("source"))

The choice between the three is mostly ergonomic. LangChain wins when you need many integrations in a hurry. LlamaIndex wins when retrieval is the centre of the system and you want first-class abstractions for hybrid retrieval, metadata filtering, and complex query engines. PyTorch + FAISS wins when the team is small, the requirements are stable, and the upgrade-cost of a rapidly-evolving framework is unattractive.

Pitfalls

Pop quiz

Interview Q&A

Interactive Demo