Capstone: Building a Production RAG System

1. When RAG beats fine-tuning — and when it does not

Use RAG when:

• Knowledge changes frequently (news, legal documents, product catalogs).
• You need source attribution — every claim must cite a specific document passage.
• Your knowledge base is large (millions of documents) and retraining is cost-prohibitive.
• You need to add private documents post-deployment without retraining.

Use fine-tuning when:

• You need to change the model’s style or format of response (RAG cannot teach tone).
• The task requires deep domain-specific reasoning that benefits from weight-level specialisation (e.g., a model that reasons like a physician, not just one that retrieves medical text).
• Latency is critical and extra retrieval + context tokens are unacceptable.
• Knowledge is stable and fits within a manageable training corpus.

2. Document pipeline: chunking strategies

The document pipeline converts raw files (PDF, HTML, Markdown, code) into chunks stored in the vector database. Chunk quality is the single biggest determinant of retrieval quality.

Fixed-size chunking

Split text into chunks of $C$ characters or $T$ tokens with overlap $O$. Simple, deterministic, fast.

def fixed_chunk(text: str, chunk_size: int = 512, overlap: int = 64) -> list[str]:
    tokens = text.split()  # word-level approximation
    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunks.append(" ".join(tokens[start:end]))
        start += chunk_size - overlap
    return chunks

Pros: simple, predictable chunk sizes (important for embedding models with token limits). Cons: splits mid-sentence, mid-table, mid-code-block — fragments semantic units.

Sentence-boundary chunking

Use a sentence tokenizer (NLTK, spaCy) to split at sentence boundaries, then group $k$ consecutive sentences into a chunk with $s$ sentences of overlap.

Pros: preserves semantic unit boundaries. Cons: short sentences create tiny chunks with little context; long sentences create large chunks that may exceed the embedding model limit.

Semantic chunking

Embed each sentence, then split wherever the cosine similarity between consecutive sentence embeddings drops below a threshold. This groups topically coherent text together regardless of raw length.

Pros: best semantic coherence; clusters related content. Cons: requires embedding all sentences upfront (slow for large corpora); threshold is a sensitive hyperparameter.

Metadata extraction: attach metadata to every chunk at ingestion time: document title, section heading, page number, creation date, author, source URL. This enables metadata filtering during retrieval (e.g., “only return chunks from documents published after 2023”), which dramatically improves precision on time-sensitive queries.

3. Embedding model choice

The embedding model converts text to dense vectors. The right choice depends on domain, language, and latency requirements.

Dense embeddings:

• text-embedding-ada-002 (OpenAI): 1536 dimensions, strong general English. Requires API call — no self-hosting.
• E5-large-v2 (Wang et al. 2022): 1024 dimensions, strong multilingual, self-hostable. Instruction-tuned variant e5-mistral-7b-instruct is state-of-the-art on MTEB as of 2024.
• BGE-large-en-v1.5 (BAAI): 1024 dimensions, state-of-the-art on BEIR for English retrieval, self-hostable.

Sparse embeddings:

• BM25: classic TF-IDF-style lexical matching. No vectors, no neural model. Excellent for exact keyword matching; fails on synonyms and paraphrase.
• SPLADE (Formal et al. 2021): learned sparse vectors from BERT, combining lexical and semantic matching. Best of both worlds but requires a GPU for inference.

Hybrid retrieval: combine dense and sparse scores at query time:

where $\alpha$ is tuned on a validation set (typically $\alpha \approx 0.7$ for dense-heavy, $\alpha \approx 0.3$ for sparse-heavy). Hybrid retrieval consistently outperforms either alone on the BEIR benchmark, which covers 18 retrieval tasks across diverse domains.

4. Vector store options

5. Retrieval strategies

Top-k retrieval

Retrieve the $k$ chunks with highest cosine similarity to the query embedding. Fast but may return redundant chunks (near-duplicates).

MMR: maximal marginal relevance

Iteratively selects chunks that are relevant to the query AND dissimilar from already-selected chunks:

where $S$ is the set of already-selected documents. MMR reduces redundancy in the retrieved set, improving diversity.

HyDE: hypothetical document embedding

HyDE (Gao et al. 2022) generates a hypothetical answer to the query using the LLM, then embeds that hypothetical answer and retrieves documents similar to it. The insight: query embeddings and document embeddings live in different regions of the embedding space (queries are short and question-shaped; documents are long and answer-shaped). A hypothetical document bridges this gap.

def hyde_retrieve(query: str, llm, embedder, vector_store, k: int = 5):
    # Step 1: generate a hypothetical document
    prompt = f"Write a detailed answer to this question: {query}"
    hypothetical_doc = llm.generate(prompt)
    # Step 2: embed the hypothetical document
    query_vector = embedder.embed(hypothetical_doc)
    # Step 3: retrieve real documents similar to the hypothetical
    return vector_store.search(query_vector, k=k)

Re-ranking with cross-encoders

First-stage retrieval (top-$k$ ANN search) is fast but approximate. A cross-encoder re-ranker scores each (query, chunk) pair jointly, producing much more accurate relevance scores at the cost of running a small BERT-class model on each pair.

from sentence_transformers import SentenceTransformer
import faiss, numpy as np

# Build index (once)
bi_encoder = SentenceTransformer("BAAI/bge-large-en-v1.5")
chunks = ["Paris is the capital of France.", "Berlin is the capital of Germany.",
        "The Eiffel Tower was built in 1889.", "Brandenburg Gate is in Berlin."]
chunk_embeddings = bi_encoder.encode(chunks, normalize_embeddings=True)
index = faiss.IndexFlatIP(chunk_embeddings.shape[1])
index.add(chunk_embeddings.astype(np.float32))

# Query
query = "When was the Eiffel Tower constructed?"
q_emb = bi_encoder.encode([query], normalize_embeddings=True)
scores, indices = index.search(q_emb.astype(np.float32), k=3)
retrieved = [chunks[i] for i in indices[0]]
print("Retrieved:", retrieved)

6. Generation: prompt construction and context management

Prompt template: structure matters. A reliable template:

You are a helpful assistant. Answer the question using only the provided context.
If the context does not contain the answer, say "I don't know."

Context:
[CHUNK 1 — source: doc_title, page 3]
...text...

[CHUNK 2 — source: doc_title, page 7]
...text...

Question: {user_question}

Answer (cite sources as [1], [2], etc. matching the chunk numbers above):

Context window management: with many retrieved chunks, you may exceed the LLM’s context window. Strategies:

• Limit total retrieved tokens (e.g., 3000 tokens for a 4096-token model).
• Use a “lost in the middle” mitigation: put the most relevant chunks at the beginning and end, not in the middle (Liu et al. 2023 found models attend poorly to middle-of-context information).
• Rerank chunks by relevance before truncating.

Citation injection: instruct the model to cite chunk IDs. Post-process the response to verify that cited chunks actually contain the claimed information (citation verification reduces hallucinated citations).

7. Evaluation with RAGAS

RAGAS [3] evaluates a RAG pipeline on four metrics that require only the question, the retrieved contexts, the generated answer, and optionally a ground-truth answer:

• Faithfulness: is every claim in the answer supported by the retrieved context? (Not supported = hallucination)
• Answer relevance: does the answer address the question? (An answer that is faithful to the context but doesn’t answer the question is irrelevant)
• Context precision: what fraction of the retrieved chunks are actually relevant to the question? (High precision = less noise in context)
• Context recall: what fraction of the ground-truth information is covered by the retrieved chunks? (High recall = nothing important missed)

from datasets import Dataset
# pip install ragas

questions = ["When was the Eiffel Tower built?", "What is the capital of Germany?"]
answers = ["The Eiffel Tower was built in 1889.", "The capital of Germany is Berlin."]
contexts = [
  ["The Eiffel Tower was constructed between 1887 and 1889 for the World's Fair."],
  ["Berlin has been the capital of Germany since reunification in 1990."]
]
ground_truths = [["The Eiffel Tower was built in 1889."], ["Berlin is the capital of Germany."]]

data = {
  "question": questions,
  "answer": answers,
  "contexts": contexts,
  "ground_truth": ground_truths,
}
dataset = Dataset.from_dict(data)

# Requires OPENAI_API_KEY or custom LLM config
# from ragas import evaluate
# from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
# result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision, context_recall])
# print(result)
# Example output: {'faithfulness': 0.97, 'answer_relevancy': 0.89, 'context_precision': 0.91, 'context_recall': 0.85}

8. Production traps

9. Practical project: RAG over ML papers

Build a question-answering system over the papers cited in this learning hub. This gives you a concrete, verifiable knowledge base where you know the ground truth.

Step 1 — Collect documents: download the PDF of each cited paper (Vaswani 2017, Wei et al. 2022, Gu & Dao 2023, Lewis et al. 2020, etc.). Extract text using pdfminer.six or pymupdf.

Step 2 — Chunk and embed: use sentence-boundary chunking with overlap 1 sentence. Embed with BGE-large-en-v1.5. Store in Chroma (local, no server needed).

Step 3 — Build retrieval pipeline: implement hybrid retrieval (BGE dense + BM25 over chunk text). Add MMR to reduce redundancy.

Step 4 — Add a cross-encoder re-ranker: use cross-encoder/ms-marco-MiniLM-L-6-v2 to re-rank top-20 candidates.

Step 5 — Wire to an LLM: use a local mistral-7b-instruct via Ollama (free, no API key) or gpt-4o-mini via OpenAI. Use the citation-injection prompt template from Section 6.

Step 6 — Evaluate: create 20 question-answer pairs from the papers. Run RAGAS. Target faithfulness above 0.90 and context recall above 0.80.

Step 7 — Add a stale-index guard: log the mtime of each source PDF alongside the chunk in Chroma metadata. On startup, check for any PDF newer than its indexed version and re-index only those.