Modern AI: Transformer Variants & Training Paradigms

Prerequisites

Transformers, Encoder, Decoder & Encoder-Decoder Models, Fine-Tuning & PEFT

Learning outcomes

Compare BERT, GPT, T5, LLaMA, and Mamba across architecture type, pretraining objective, and best-fit tasks.
Compare fine-tuning methods (full FT, LoRA, QLoRA, prefix tuning, RLHF, DPO) on compute cost, quality, and memory.
Choose between RAG, fine-tuning, and in-context learning based on problem requirements.
Explain what LoRA assumes about weight updates and why this enables parameter efficiency.
Describe the mathematical insight behind DPO working without a separate reward model.

TL;DR

The modern LLM landscape splits into encoder-only (BERT for understanding), decoder-only (GPT/LLaMA for generation), and encoder-decoder (T5 for seq2seq). Mamba challenges Transformers with linear-time SSM-based modeling. Fine-tuning choice depends on compute budget: full FT for maximum quality, LoRA/QLoRA for efficiency, DPO for preference alignment without RL. RAG, fine-tuning, and in-context learning address complementary needs.

Architecture Comparison: BERT vs GPT vs T5 vs LLaMA vs Mamba

Model	Architecture	Pretraining objective	Context handling	Best for	Parameter efficiency	Open weights?
BERT	Encoder-only Transformer	Masked language modeling (MLM) + NSP	Bidirectional full attention — all tokens see all tokens	Classification, NER, QA (discriminative NLP)	Medium — large model for classification heads	Yes (Google)
GPT-2/3	Decoder-only Transformer	Causal language modeling (next-token prediction)	Left-to-right causal attention	Text generation, few-shot prompting, in-context learning	Excellent scaling; few-shot with no fine-tuning	Partial (GPT-2 yes; GPT-3/4 no)
T5	Encoder-decoder Transformer	Span corruption (denoising) + text-to-text framing	Encoder: bidirectional; decoder: causal	Translation, summarisation, any seq2seq task	Good — one model for all NLP via task prefix	Yes (Google)
LLaMA 2/3	Decoder-only Transformer	Causal LM (like GPT); RoPE positional encoding, GQA	Causal; extended context via RoPE	Open-source generation, instruction following, fine-tuning base	Excellent — open weights enable wide PEFT	Yes (Meta)
Mamba	Selective State Space Model (SSM) — no attention	Causal LM (same as GPT training)	O(L) in time and memory; selective gating replaces attention	Long sequences, streaming, memory-constrained inference	High — fixed state size regardless of sequence length	Yes (Gu & Dao)

Fine-Tuning Methods Comparison

Method	Trainable params	Memory	Quality	When to use
Full Fine-Tuning	All (~100%)	Very high (all gradients + optimiser states)	Best possible for task	Abundant compute, unique task distribution, maximise benchmark performance
LoRA	Low-rank adapters only (~0.1–1%)	Low (adapter grads only; frozen base)	Near full-FT on most tasks	GPU-memory-constrained fine-tuning; multi-task serving (swap adapters); most practical default
QLoRA	LoRA adapters on 4-bit quantised base	Very low (4-bit base + adapter grads in BF16)	Slightly below LoRA (quantisation noise)	Single GPU fine-tuning of 70B+ models; consumer hardware
Prefix Tuning	Prepended virtual tokens per layer (~0.1%)	Low	Moderate (less expressive than LoRA)	When model weights must be frozen entirely (e.g., API-only access)
Prompt Tuning	Soft input tokens only (~0.01%)	Minimal	Weaker than LoRA; works at large scale	Very large frozen models (>11B) where even LoRA is too expensive
RLHF (PPO)	Policy (full or LoRA) + reward model	Very high (4 models: ref, policy, RM, value)	Excellent for alignment and instruction following	Safety alignment, helpfulness tuning; ChatGPT-style RLHF pipeline
DPO	Policy only (full or LoRA, no RM)	Medium (2 models: policy + frozen ref)	Competitive with PPO; simpler	Preference alignment without RL complexity; open-source RLHF alternative

RAG vs Fine-Tuning vs In-Context Learning

Method	Knowledge update mechanism	Compute at inference	Good for	Bad for
In-Context Learning (ICL)	Append examples or docs in prompt	Proportional to prompt length	Quick prototyping; few-shot tasks; when no fine-tuning budget	Long documents (context limit); consistent formatting; cost at scale
RAG	Retrieve relevant docs from external index at query time	Retrieval + LLM inference	Frequently updated knowledge; cited factual QA; long-tail queries	Complex multi-hop reasoning; latency-sensitive; when knowledge is stable
Fine-Tuning	Embed knowledge into weights via gradient updates	Standard LLM inference (no retrieval)	Stable knowledge, new writing style, task-specific format, reduced hallucination on common facts	Rapidly changing facts; rare knowledge that is poorly represented in training data

The 2023-2025 practitioner's decision tree

Start here: Is the task about factual knowledge that changes frequently or is too large for fine-tuning? → RAG.

Otherwise: Do you have preference-labeled data or need alignment? → DPO (simpler) or RLHF (more complex, possibly higher quality).

Otherwise: Do you need to adapt to a new task/style with limited examples? → LoRA (default) or QLoRA (single GPU). Full fine-tuning only when you have abundant data and compute.

Quick baseline: If the task fits in context and you’re prototyping → In-context learning first.

Pop Quiz

Check yourself

Score: 0 / 5(0 answered)

1. BERT uses bidirectional attention while GPT uses causal (left-to-right) attention. What is the consequence for their best use cases?
2. LoRA assumes that fine-tuning weight updates have low intrinsic rank. What does this mean practically?
3. When would you choose RAG over fine-tuning for a QA application?
4. Why does DPO work without a reward model — what is the mathematical insight?
5. Mamba achieves linear-time sequence modeling (O(L)) unlike Transformers (O(L²)). What is the core mechanism?

Interview Q&A

LoRA vs full fine-tuning — what does LoRA assume about weight updates?onsite

LoRA [7] is based on the intrinsic dimensionality hypothesis: the weight update $\Delta W$ needed for task adaptation has low intrinsic rank. Formally, instead of learning a full $d \times k$ update matrix, LoRA parameterises $\Delta W = B A$ where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ with $r \ll \min(d, k)$ . Only $B$ and $A$ are trained; $B$ is initialised to zero so $\Delta W = 0$ at the start (preserving the pretrained output exactly). At inference, $W + BA$ is a single merged matrix — no additional compute.

Empirically, ranks as low as $r = 4$ – $16$ match full fine-tuning on most tasks. Full fine-tuning requires gradients and optimiser states for every parameter (typically 2–3x memory per param); LoRA reduces trainable parameters by 100x–10000x.

Full fine-tuning wins when: (1) the adaptation is fundamentally high-rank (large distribution shift), (2) you have abundant data and the task requires overfitting to specific patterns, or (3) adapter merging would lose multi-task flexibility.

When would you use RAG instead of fine-tuning — and when is fine-tuning better?onsite

Use RAG when:

Knowledge is updated frequently (news, product catalogs, medical literature) — retraining weights is too slow and expensive.
The knowledge base is too large to bake into weights reliably.
Source attribution is required — RAG naturally provides cited passages.
You need precise factual recall for long-tail queries not well-covered in pretraining.

Use fine-tuning when:

You need a specific output style, tone, or format that is hard to achieve with prompting.
The task has stable, bounded knowledge that can be expressed in weights.
Low-latency inference is critical — no retrieval step.
The number of unique task examples is manageable and the distribution is well-defined.

Use both (RAG + fine-tuning): when you need both style adaptation (fine-tune) and up-to-date factual grounding (RAG). This is the architecture used in production knowledge assistants.

Why does DPO work without a reward model — what is the mathematical insight?staff

The RLHF KL-regularised objective has a closed-form optimal solution:

\pi^*(y|x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y|x) \exp\!\left(\frac{r(x,y)}{\beta}\right)

Inverting this:

r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)

The reward is expressed entirely in terms of the optimal policy and the partition function. Substituting into the Bradley-Terry preference model:

p(y_w \succ y_l) = \sigma\!\left(\beta \log \frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)

The intractable $Z(x)$ terms cancel. Replacing $\pi^*$ with the trainable $\pi_\theta$ gives the DPO loss — a binary cross-entropy over log-ratio differences. The policy $\pi_\theta$ simultaneously plays the role of the reward model (through its ratio to $\pi_{\text{ref}}$ ) and the fine-tuned language model. This works because the RLHF optimal policy already implicitly encodes the reward function in its deviation from the reference policy.

References

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017 (arXiv preprint). arXiv:1706.03762
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019 (arXiv preprint). arXiv:1810.04805
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language Models are Few-Shot Learners. NeurIPS 2020 (arXiv preprint). arXiv:2005.14165
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 21(140), 1-67 (arXiv preprint). arXiv:1910.10683
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint. arXiv:2307.09288
Gu, A., and Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv preprint. arXiv:2312.00752
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022 (arXiv preprint). arXiv:2106.09685
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023 (arXiv preprint). arXiv:2305.14314
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 (arXiv preprint). arXiv:2305.18290
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020 (arXiv preprint). arXiv:2005.11401

History of AI & ML