Fine-Tuning, LoRA & PEFT, RLHF, DPO

Background

Full fine-tuning of LLMs requires updating all parameters (billions) - too expensive for most organizations
LoRA (Hu et al. 2021) reduced adapter parameters by 10,000x while matching full fine-tune performance
PEFT (Parameter-Efficient Fine-Tuning) is now the standard for adapting LLMs to downstream tasks

The Concept

Fine-tuning adapts a pretrained model to a specific task by continuing training on labeled task data. PEFT methods like LoRA add a small set of trainable parameters while freezing the main model, achieving 90%+ of full fine-tune quality with <1% of the parameters updated.

How it Links

LoRA decomposes weight updates into low-rank matrices - connects to SVD and matrix factorization
QLoRA (4-bit quantized LoRA) makes fine-tuning feasible on consumer hardware
RLHF (instruction tuning) typically uses LoRA adapters to align base models with human preferences

Concrete Example

Adapting GPT-style base model to a medical Q&A task: full fine-tune requires 40GB GPU memory for 7B parameters. LoRA adds two small matrices per attention layer (rank=8), training only 4 million extra parameters. The resulting model performs comparably on medical benchmarks at 1/1000th the compute.

Prerequisites

The Transformer Architecture, Encoder, Decoder & Encoder-Decoder Models, Scaling Laws & Modern Training

Learning outcomes

Contrast full fine-tuning with parameter-efficient fine-tuning (adapters, prefix tuning, prompt tuning, LoRA, QLoRA) on trainable parameters, memory, training speed, quality, and whether the update is mergeable at inference.
Derive the LoRA rank-r update $W^{\prime} = W_0 + (\alpha / r)\, B A$ with $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$ and $r \ll \min(d, k)$, and explain why training touches only $B$ and $A$ while inference can fold the update back into a single matrix.
Explain QLoRA: 4-bit NF4 block-wise quantization of the frozen base, double quantization of the quantization constants, and paged optimizers, and why this lets a 65B model fine-tune on a single 48 GB GPU.
Walk through the RLHF pipeline end to end: SFT, Bradley-Terry reward model, and PPO with a KL penalty against a fixed reference policy, and state the full PPO objective $J(\theta) = \mathbb{E}\left[r_\phi(x, y) - \beta\, \mathrm{KL}(\pi_\theta(y \mid x)\,\|\,\pi_\mathrm{ref}(y \mid x))\right]$.
Derive DPO from the closed-form RLHF optimum under a Bradley-Terry preference model and a KL constraint, recovering $\mathcal{L}_\mathrm{DPO} = -\mathbb{E}\left[\log \sigma\!\left(\beta \log \tfrac{\pi(y_w \mid x)}{\pi_\mathrm{ref}(y_w \mid x)} - \beta \log \tfrac{\pi(y_l \mid x)}{\pi_\mathrm{ref}(y_l \mid x)}\right)\right]$ and explain why this eliminates the explicit reward model.

TL;DR

Full fine-tuning is rarely worth it once base models cross a few billion parameters: adapters [1], prefix tuning [2], prompt tuning [3], and especially LoRA [4] and QLoRA [5] freeze the base and train a tiny update that is mergeable at inference. Instruction tuning [6][7] teaches the model to follow tasks; RLHF [8][9] aligns it to human preferences via SFT, a Bradley-Terry reward model, and PPO with a KL penalty against a frozen reference policy; DPO [10] derives a closed-form preference loss that drops the reward model entirely; Constitutional AI [11] swaps human feedback for AI-generated critiques.

Intuition

A pretrained LLM is a very expensive blob of weights. Full fine-tuning treats those weights as a starting point and updates all of them on a downstream task; a 7B model in mixed-precision Adam needs roughly $140$ GB of optimizer state (see Scaling Laws & Modern Training), so even a “small” full fine-tune wants a multi-GPU node. Worse, every task gets its own full-size copy of the weights, which is unworkable once you want to ship dozens of specialized models.

The modern default is the opposite picture: freeze the base, train a small update. The intuition is that fine-tuning a strong pretrained model usually just nudges its behavior, so the change to the weights lives on a low-dimensional manifold inside the full parameter space [4]. Adapters [1] express that update as small bottleneck modules inserted between layers; prefix and prompt tuning [2][3] express it as learned input vectors; LoRA [4] expresses it as a low-rank decomposition $\Delta W = B A$ that you can fold back into the frozen weights at inference. QLoRA [5] stacks 4-bit base quantization on top of LoRA so a 65B model fine-tunes on a single $48$ GB GPU. The same picture extends past task adaptation into preference alignment: RLHF [9] and DPO [10] both update the policy on top of a frozen reference, regularized so it does not drift far from where pretraining left it [12].

Fine-tuning, in one sentence

Pretraining gave you a model that knows the world; PEFT lets you teach it a job without rewriting what it already knows. You freeze the $\sim 7$ billion base parameters, slot in a few million trainable parameters that act as a small “delta” on top of the base, and only those tiny parameters move during training. The base model is shared across every task you train for, so each new task costs megabytes instead of gigabytes; and for methods like LoRA the delta can be merged back into the base at inference time so there is zero runtime overhead versus the original model.

2019Houlsby adapterspaper
First parameter-efficient transfer-learning recipe for Transformers. Insert small bottleneck MLPs between layers, train only those, match BERT full-FT quality on GLUE with about 3% of the parameters trainable.
2021Prefix tuning (Li and Liang)paper
Instead of editing weights, prepend a learned continuous prefix to the keys and values of every attention layer. Strong on generation tasks at GPT-2 / BART scale; trainable parameters scale only with prefix length.
2021Prompt tuning (Lester)paper
Even simpler: learn a soft prompt at the input embedding only. Closes the gap to full fine-tuning as the base model crosses about 10B parameters; below that it lags adapters and prefix tuning.
2021LoRApaper
Express the weight delta as a rank-r product B A with r much smaller than d, train only B and A, merge back into W at inference. Becomes the de-facto PEFT method for LLMs in 2023.
2022RLHF and InstructGPTrelease
OpenAI applies RL-from-human-preferences at LLM scale: SFT on demonstrations, Bradley-Terry reward model on preference pairs, PPO with a KL penalty against the SFT reference. Turns GPT-3 into ChatGPT.
2023QLoRApaper
Quantize the frozen base to 4-bit NF4, double-quantize the quantization constants, train LoRA adapters on top, swap in paged optimizers. Fine-tunes 65B-parameter models on a single 48 GB GPU at full-precision quality.
2023DPOpaper
Closed-form derivation of the RLHF optimum under a Bradley-Terry preference model collapses the SFT-RM-PPO pipeline into one supervised loss on preference pairs. No reward model, no rollouts, no PPO.

Full fine-tuning vs PEFT: the parameter-efficient family

Full fine-tuning sits at one extreme: every parameter is trainable, the optimizer state mirrors the weight tensor on every device, and you pay the same memory and compute as a small pretraining run. The PEFT family lives at the other extreme: the base is frozen and untouched, and only a thin slice of additional parameters (a fraction of one percent of the base) moves during training. The point of comparison is not just “how many parameters do you train”, but how the method affects memory, training speed, downstream quality, and whether the learned update can be merged back into the base weights for inference.

The numbers below assume a $7$ B-parameter base in BF16. Memory cost is dominated by Adam optimizer state on the trainable parameters (about $12$ bytes per trainable parameter for FP32 master weights, momentum, and variance) plus the frozen base weights themselves. “Mergeable at inference?” answers whether you can fold the trained update back into a single matrix, so deployment cost is identical to the original model.

Method	Trainable params (7B base)	Memory cost	Training speed	Quality vs full FT	Mergeable at inference?
Full FT	~7B (100%)	~140 GB (FP16 weights + FP32 Adam state)	1x baseline; throughput limited by full backward + optimizer step	Reference baseline by definition	N/A (no separate update)
Adapters (Houlsby)	~30-200M (0.5-3%)	~16-18 GB (frozen base + small adapter optimizer state)	~1.5-2x faster step than full FT; extra forward latency from inserted bottleneck MLPs	Within ~0.5% on GLUE / SuperGLUE at BERT-large scale	No: adapters add new layers in the forward path
Prefix tuning	~1-10M (<0.1%)	~15 GB; prefix tensors are tiny	~2x faster than full FT; small overhead for prefix concat in attention	Strong on generation, weaker than LoRA on classification at <10B scale	No: prefix is concatenated into K/V at every layer
Prompt tuning	~100K-1M (<0.01%)	~14 GB; only an input-embedding-sized prompt	~2-3x faster than full FT; near-zero per-step adapter overhead	Lags adapters / prefix below 10B base; closes the gap above 10B	No: prompt is concatenated to the input sequence
LoRA (r=16)	~4M (~0.06%)	~30 GB (FP16 frozen base + tiny LoRA optimizer state)	~2x faster than full FT; backward only flows through B and A	Within ~1% of full FT on most benchmarks at 7B-70B	Yes: W_merged = W_0 + (alpha / r) * B * A; zero inference overhead
QLoRA (NF4 + r=16)	~4M (~0.06%)	~20 GB (4-bit NF4 base + LoRA optimizer state + paged Adam)	~1.5x faster than full FT; dequant overhead in forward, paged optimizer hides spikes	Statistically indistinguishable from full FT on Vicuna / MMLU at 7B-65B	Yes but lossy: merging requires dequantizing the base back to FP16 first

LoRA: low-rank adaptation

LoRA: rank-decomposition update

Hu et al. [4] start from the empirical observation that fine-tuning updates to a strong pretrained model are approximately low-rank: the change $\Delta W$ to a $d \times k$ weight matrix concentrates almost all of its Frobenius norm in the top few singular directions. Instead of training the full $\Delta W \in \mathbb{R}^{d \times k}$ , LoRA parameterizes it as a rank- $r$ product.

Let $W_0 \in \mathbb{R}^{d \times k}$ be a frozen pretrained weight matrix. Write the update as

\Delta W = B A, \qquad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}, \quad r \ll \min(d, k).

The forward pass through the adapted layer is

h = W_0 x + \frac{\alpha}{r}\, B A x,

where $\alpha$ is a tunable scaling factor that decouples the chosen rank from the effective learning rate on the LoRA branch. Only $A$ and $B$ are trainable; $W_0$ never receives a gradient.

Initialization. Sample $A$ entry-wise from a small Gaussian (the original paper uses Kaiming-uniform on $A$ ) and set $B = 0$ . The product $B A$ is therefore exactly zero at step 0, so the LoRA-augmented model produces identical outputs to the frozen base before any training. Training is a pure additive perturbation that grows out of zero, which removes any “warmup shock” from a randomly initialized adapter.

Trainable parameter count. The full update would have $d k$ parameters. The LoRA factorization has

|A| + |B| = r k + r d = r (d + k)

parameters. For a typical Transformer projection at $d = k = 4096$ and $r = 16$ :

r (d + k) = 16 \cdot 8192 = 131072, \qquad d k = 4096 \cdot 4096 = 16777216,

a saving factor of $16777216 / 131072 = 128\times$ on this single matrix. The same saving compounds across every projection you LoRA-fy in the model.

Inference: merge once, serve normally. Because $\Delta W = (\alpha / r) B A$ is just a matrix of the same shape as $W_0$ , you can compute

W_{\mathrm{merged}} = W_0 + \frac{\alpha}{r}\, B A

once after training, replace $W_0$ in memory with $W_{\mathrm{merged}}$ , and serve a single matrix. There is zero runtime overhead versus the original model: no extra matmuls, no extra activations, no extra memory traffic. This is the property that makes LoRA dominant for production deployment; adapters and prefix / prompt tuning all leave residual structure in the forward path.

Which Transformer matrices to LoRA. The original LoRA paper recommended adapting just the query and value projections ( $W_Q$ , $W_V$ ) inside attention, finding that this matched or beat full fine-tuning on GLUE / WikiSQL / SAMSum at small $r$ . Modern recipes are more aggressive: production fine-tunes (the Hugging Face PEFT defaults, the QLoRA configs, the LLaMA-2-Chat recipe) typically LoRA all four attention projections ( $W_Q, W_K, W_V, W_O$ ) plus the two FFN projections ( $W_{\mathrm{up}}$ , $W_{\mathrm{down}}$ , and gate for SwiGLU variants). The extra trainable parameters are still a tiny fraction of the base, and the quality lift on harder downstream tasks is consistent.

The role of the alpha hyperparameter. The factor $\alpha / r$ in the forward pass looks redundant; you could absorb it into $B$ or $A$ . The reason it is exposed is that it lets you change $r$ without retuning the learning rate. If you double $r$ , the natural variance of $B A$ at initialization roughly doubles too, so the per-step update on the LoRA branch is larger. Setting $\alpha = 2 r$ (a common default) or $\alpha = r$ keeps the effective scale of the update roughly constant across $r$ values, so a hyperparameter sweep over $r \in \{4, 8, 16, 32, 64\}$ uses the same learning rate. In practice $\alpha$ is set once per recipe (often $\alpha = 2 r$ ) and not retuned per task.

QLoRA: 4-bit base + LoRA on top

LoRA cuts the optimizer-state line item to a rounding error, but it does nothing about the base weights themselves: a 65B model in BF16 still needs $130$ GB just to hold $W_0$ , before any activations or LoRA adapters. Dettmers et al. [5] close the remaining gap by quantizing the frozen base to 4 bits and stacking three engineering tricks on top so the quality holds up.

The first trick is 4-bit NF4 quantization, a non-uniform 4-bit format calibrated for normally-distributed weights. Pretrained weight tensors are empirically close to a zero-mean Gaussian, so the optimal $16$ -level codebook for them is not equally spaced (INT4) and not exponent-driven (FP4); it is a set of levels chosen so that each codebook entry gets roughly equal probability mass under a unit Normal. NF4 uses exactly that codebook, applied block-wise with block size $64$ : the tensor is chopped into blocks of $64$ elements, each block gets its own absmax-based scale, and the cost of one outlier in the tensor only contaminates the $64$ elements that share its block. NF4 outperforms FP4 and INT4 at the same $4$ -bit budget on every quality metric Dettmers measured, because the codebook matches the empirical distribution of the weights it has to represent.

The second trick is double quantization. Each block of $64$ weights needs its own FP32 scale, which costs $32 / 64 = 0.5$ bits per parameter on top of the 4 bits of weight payload. Dettmers quantizes those per-block scales themselves to 8 bits (with their own coarser scale), saving roughly $0.37$ bits per parameter at no measurable quality loss. The total budget per base parameter lands close to $4.1$ bits instead of the naive $4.5$ , which matters a lot at 65B scale.

The third trick is paged optimizers. Even with 4-bit base weights and tiny LoRA adapters, the optimizer state on the LoRA-trainable parameters is still FP32, and during training it can briefly compete with activations for GPU memory (a long-sequence forward, a checkpointed backward, an outlier batch). NVIDIA Unified Memory lets the driver page that FP32 Adam state out to CPU RAM when GPU memory pressure spikes, and back in for the optimizer step. The cost is some PCIe traffic; the win is the elimination of hard OOMs on borderline runs.

Stack these three and the memory budget for a 65B fine-tune comes out to: 4-bit NF4 base $\approx 41$ GB (vs $130$ GB at FP16), LoRA adapters across all attention and FFN projections at $r = 64$ add another $\approx 1$ GB, and the FP32 Adam state on the LoRA-only trainable parameters is in the low single-digit GB. Total: about $48$ GB, which fits a single A6000 or A40. Quality on the Vicuna benchmark and MMLU matches a full BF16 fine-tune within statistical error [5], which is the result that flipped open-weights fine-tuning from “needs a node” to “needs a workstation” essentially overnight.

Instruction tuning: FLAN and FLAN-T5

LoRA and QLoRA address the how of fine-tuning; FLAN [6] addresses the what. Wei et al. observed that supervised fine-tuning on a mixture of NLP tasks, each templated as “instruction + input -> output”, gives the model strong zero-shot generalization to held-out task types. The trained model is not just better at the tasks in the mixture; it follows instructions for tasks it has never seen, because it has learned the meta-task of “do what the instruction says”.

FLAN-T5 [7] scaled this recipe. Chung et al. mixed $1{,}836$ tasks (drawn from Muffin, T0-SF, NIv2, and a chain-of-thought set), added explicit chain-of-thought examples so the model learns to verbalize multi-step reasoning, and applied the procedure across the full T5 model family up to T5-XXL. Two findings dominated the paper. First, instruction tuning combines additively with model scale: FLAN-T5-XXL is roughly $10\%$ better than T5-XXL on MMLU, and the gap holds at smaller sizes too, so you get the lift on top of whatever pretraining bought you. Second, the procedure is essentially free at inference time: there is no extra latency, no extra parameters at serve time, no separate model to ship; you just deploy the instruction-tuned weights instead of the base.

The composition of the training mixture matters a lot. Too narrow and the model loses general capability (it overfits to the supervised tasks at the expense of pretraining knowledge); too broad and tasks compete for capacity (the model becomes mediocre at everything rather than competent at the families the mixture emphasizes). Modern instruction-tuned releases (LLaMA-2-Chat, Mistral-Instruct, Qwen-Chat) all use a recipe that is recognizably descended from FLAN: a curated, deduplicated multi-task instruction corpus, often combined with a smaller chain-of-thought set, applied as supervised fine-tuning on top of the pretrained base before any preference-alignment stage.

RLHF: SFT, reward model, PPO with KL

The RLHF pipeline: SFT, reward model, PPO with KL

The RLHF recipe has three stages, each of which builds on the previous one.

Stage A: supervised fine-tuning (SFT). Start from a pretrained base. Fine-tune on a curated set of high-quality (prompt, response) demonstrations. The result is the SFT policy $\pi_{\mathrm{SFT}}$ . This both teaches a baseline behavior and gives you the reference policy $\pi_{\mathrm{ref}} = \pi_{\mathrm{SFT}}$ used in the next two stages.

Stage B: reward model. Collect preference pairs: for each prompt $x$ , sample two responses $y_w, y_l$ from $\pi_{\mathrm{SFT}}$ and have a human label which is preferred. Fit a reward model $r_\phi(x, y)$ under the Bradley-Terry preference model:

P(y_w \succ y_l \mid x) = \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)).

Train $r_\phi$ by maximum likelihood on the labeled pairs:

\mathcal{L}_{\mathrm{RM}}(\phi) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\,[\log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l))].

In practice $r_\phi$ is the SFT model with its language-modeling head replaced by a scalar regression head; the rest of the weights are inherited from SFT. Reward modeling gives you a learned proxy for “what humans prefer”.

Stage C: PPO with KL. Now optimize the policy to maximize expected reward, regularized so it does not drift far from the reference. The objective is:

J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)}\!\left[r_\phi(x, y) - \beta\, \mathrm{KL}(\pi_\theta(\cdot \mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot \mid x))\right].

The KL penalty $\beta\, \mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}})$ is the load-bearing term. Without it the policy will collapse to mode-seeking sequences that maximize the (imperfect) reward model: it will discover what is sometimes called reward hacking and produce gibberish that scores well under $r_\phi$ but is unusable. With it, the policy is constrained to the neighborhood of $\pi_{\mathrm{ref}}$ , so it can only spend its capacity to nudge behavior in the directions reward suggests.

PPO itself adds a clipped importance-sampling objective on top. For action $a_t = y_t$ at step $t$ , define the per-token probability ratio $\rho_t(\theta) = \pi_\theta(y_t \mid x, y_{<t}) / \pi_{\theta_{\mathrm{old}}}(y_t \mid x, y_{<t})$ where $\theta_{\mathrm{old}}$ is the policy that generated the rollout. The clipped surrogate is:

\mathcal{L}^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min(\rho_t(\theta) \hat A_t,\, \mathrm{clip}(\rho_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat A_t)\right],

with the typical clip range $\epsilon = 0.2$ . The clip bounds how much one update step can move the policy on any single token; without it, a few high-advantage rollouts will yank the policy far from $\theta_{\mathrm{old}}$ and the trust-region assumption (importance weights still close to $1$ ) breaks. The advantage estimate $\hat A_t$ is computed via GAE on the per-token reward $r_\phi(x, y_{1:t}) - \beta\, \mathrm{KL}$ .

The full RLHF training loop alternates: rollout from $\pi_\theta$ , score with $r_\phi$ , compute advantages, run a few PPO epochs on the clipped objective, and repeat. Ouyang et al. [9] applied exactly this recipe to GPT-3 to produce InstructGPT, which is the technical lineage of ChatGPT. The original idea (RL from human preferences) goes back to Christiano et al. [8].

DPO: preference optimization without a reward model

DPO derives from the RLHF closed form

The starting point is the same constrained optimization that PPO solves implicitly. Maximize expected reward subject to a KL constraint:

\max_{\pi}\, \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi(\cdot \mid x)}[r(x, y)] \quad \text{s.t.} \quad \mathrm{KL}(\pi(\cdot \mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot \mid x)) \le \delta.

Equivalently, the Lagrangian with multiplier $\beta > 0$ is:

\max_{\pi}\, \mathbb{E}\!\left[r(x, y) - \beta \log \tfrac{\pi(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}\right].

Maximize pointwise over $\pi(\cdot \mid x)$ subject to the simplex constraint $\sum_y \pi(y \mid x) = 1$ . The first-order condition gives the closed form:

\pi^{*}(y \mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y \mid x)\, \exp\!\left(\frac{1}{\beta} r(x, y)\right), \qquad Z(x) = \sum_{y'} \pi_{\mathrm{ref}}(y' \mid x)\, \exp\!\left(\frac{1}{\beta} r(x, y')\right).

This is a standard result; the optimal KL-regularized policy is the reference policy reweighted by $\exp(r / \beta)$ . Now invert it. Solve for $r(x, y)$ in terms of the optimal policy:

r(x, y) = \beta \log \frac{\pi^{*}(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x).

The key observation, from Rafailov et al. [10], is that any reward function $r$ that produces a given $\pi^{*}$ via the RLHF formula can be expressed in this form, and the partition function $\beta \log Z(x)$ is a function of $x$ only. Plug this expression for $r$ into the Bradley-Terry preference model:

P(y_w \succ y_l \mid x) = \sigma(r(x, y_w) - r(x, y_l)).

The two $\beta \log Z(x)$ terms are identical because they depend only on $x$ , so they cancel in the difference $r(x, y_w) - r(x, y_l)$ :

P(y_w \succ y_l \mid x) = \sigma\!\left(\beta \log \frac{\pi^{*}(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \beta \log \frac{\pi^{*}(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}\right).

The reward model has been eliminated. The Bradley-Terry preference probability is now expressed entirely in terms of the policy ratios $\pi / \pi_{\mathrm{ref}}$ . Maximizing the log-likelihood of the preference data over $\pi$ directly gives the DPO loss:

\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\!\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}\right)\right].

This is just a binary cross-entropy loss on a logit that is a difference of two log-likelihood ratios. No reward model. No rollouts. No PPO. No advantage estimator. Training reduces to a supervised fine-tune on the preference pairs, with the reference policy held frozen.

Three engineering implications. First, DPO is dramatically simpler to implement and debug than RLHF: there is no separate reward-model training, no reinforcement-learning loop, no on-policy rollouts to manage. Second, $\beta$ plays the same role as the KL coefficient in RLHF; small $\beta$ is more aggressive (the loss tolerates large policy moves), large $\beta$ is more conservative. The typical setting is $\beta \in [0.1, 0.5]$ . Third, DPO inherits whatever the SFT distribution gave you: it shifts probability mass from rejected toward chosen responses, but it cannot put mass on responses that $\pi_{\mathrm{SFT}}$ never generates. SFT data quality remains the upstream bottleneck.

Constitutional AI and RLAIF

Bai et al. [11] at Anthropic asked: can the human feedback step be replaced by AI feedback? Their Constitutional AI recipe answers yes, with two stages.

Stage 1 (supervised, “critique-and-revise”). Start from an SFT model that already follows instructions. Sample a response to a prompt that is likely to elicit harmful behavior. Then have the model self-critique its response against a written “constitution” (a list of principles like “be helpful but not harmful”), and revise. The (prompt, revised-response) pairs become a new SFT dataset, and the model is fine-tuned on them.

Stage 2 (preference alignment, RLAIF). Sample two responses for each prompt and have the model itself score which is preferred under the constitution. The preference pairs go into the same Bradley-Terry reward-model training as RLHF; the rest of the pipeline (PPO with KL or DPO) is identical, just with AI-generated preferences instead of human ones.

Why bother. The human-feedback bottleneck in RLHF is real: collecting hundreds of thousands of high-quality preference labels is slow, expensive, and labelers disagree more than the literature usually admits. An AI labeler with a written constitution is cheap, fast, and consistent. Anthropic has reported that Constitutional AI matches RLHF on harmlessness while being substantially cheaper to train. The trade-off is that the AI labeler inherits whatever biases the underlying base model has, which the constitution can patch only partially.

Full FT vs LoRA vs QLoRA on a 7B base

The previous sections laid out the asymptotics; the practical question is what those asymptotics look like on the recipe most teams actually run, which is a $7$ B base in BF16 with a batch on the order of $2$ k tokens. The answer comes from three sources: the trainable-parameter counts in the LoRA paper [4], the QLoRA memory measurements [5], and a standard mixed-precision Adam accounting of about $20$ bytes per trainable parameter (FP16 weights + FP16 grads + FP32 master + FP32 $m$ + FP32 $v$ ).

The numbers below should be read as ballpark figures, not benchmarks. Step time is relative because the absolute number depends on the kernel mix on your hardware, and quality is “within $X\%$ ” because the variance across tasks and seeds in the original papers is itself a few tenths of a percent. The interesting structure is the practical-home column: a full BF16 fine-tune wants a node, vanilla LoRA fits on a single $80$ GB GPU, and QLoRA pushes 7B to a $24$ GB consumer card and 65B to a single $48$ GB workstation card.

Recipe	Trainable params	Peak GPU memory (BF16, 7B base, batch ~2k tokens)	Step time (relative)	MMLU 5-shot vs full-FT baseline	Practical home
Full BF16 fine-tune	7.0B (100%)	~140 GB across 8x 80GB GPUs (FSDP ZeRO-3) + activations	1.0x	baseline	8x H100 / A100 node
LoRA r=16, all attn + FFN	~6M (~0.09%)	~28 GB on 1x 80GB (FP16 frozen base + BF16 acts + tiny LoRA optimizer state)	~0.6x (lighter backward)	within ~0.5% on average across tasks	1x A100/H100 80GB
LoRA r=64, all attn + FFN	~24M (~0.34%)	~32 GB on 1x 80GB	~0.7x	within ~0.3% on average	1x A100/H100 80GB
QLoRA NF4 + r=16	~6M (~0.09%)	~16 GB on 1x 24GB (NF4 base + paged Adam)	~0.8x (dequant overhead in fwd)	within ~0.5% on average; matches FP16 LoRA	1x RTX 3090/4090 24GB
QLoRA NF4 + r=64 on a 65B base	~80M (~0.12%)	~48 GB on 1x 48GB	~0.9x	within ~1% on Vicuna / MMLU	1x A6000/A40 48GB

Wiring it up: PEFT, TRL DPO, TRL PPO

The 2024 production stack for everything in this chapter is small: Hugging Face PEFT (PEFTConfig plus get_peft_model plus the standard Trainer) for LoRA fine-tuning, TRL’s DPOTrainer for preference alignment without a reward model, and TRL’s PPOTrainer for the full RLHF loop. The three differ in what dataset they consume (instruction-formatted text vs preference pairs vs prompts plus a reward model) and which trainer they expose, but they all sit on top of the same Hugging Face Transformers and Accelerate plumbing, so the surrounding code (tokenizer, data collator, distributed setup) is essentially identical across the three.

python

import torch
from transformers import (
  AutoModelForCausalLM,
  AutoTokenizer,
  BitsAndBytesConfig,
  Trainer,
  TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
  load_in_4bit=True,
  bnb_4bit_quant_type="nf4",
  bnb_4bit_compute_dtype=torch.bfloat16,
  bnb_4bit_use_double_quant=True,
)

base = AutoModelForCausalLM.from_pretrained(
  "meta-llama/Llama-2-7b-hf",
  quantization_config=bnb,
  device_map="auto",
)
base = prepare_model_for_kbit_training(base)

lora_cfg = LoraConfig(
  r=16,
  lora_alpha=32,
  lora_dropout=0.05,
  bias="none",
  task_type="CAUSAL_LM",
  target_modules=[
      "q_proj", "k_proj", "v_proj", "o_proj",
      "gate_proj", "up_proj", "down_proj",
  ],
)
model = get_peft_model(base, lora_cfg)
model.print_trainable_parameters()  # ~0.09% of base

args = TrainingArguments(
  output_dir="out",
  per_device_train_batch_size=4,
  gradient_accumulation_steps=8,
  learning_rate=2e-4,
  bf16=True,
  num_train_epochs=3,
)
trainer = Trainer(model=model, args=args, train_dataset=chat_dataset)
trainer.train()

PEFT registers the LoRA adapters as trainable nn.Linear modules over a frozen base; Trainer.train() updates only those.

Pitfalls

Over-large LoRA rank wastes the saving

The original LoRA paper found that $r = 4$ to $r = 16$ captures essentially all the lift on most tasks; pushing $r$ toward $d$ gains nothing on quality and erases the point of LoRA. If your run is not converging at $r = 16$ , the problem is almost always learning rate, the chosen target modules, or SFT data quality, not rank. [4]

Forgetting the reference model in RLHF / DPO

Without the KL penalty against $\pi_{\mathrm{ref}}$ (in RLHF) or the $\log(\pi_\theta / \pi_{\mathrm{ref}})$ parameterization (in DPO), the policy collapses to mode-seeking sequences that maximize the imperfect reward. The output is gibberish that scores well under $r_\phi$ but is useless. The fix is mechanical: never run preference alignment without a frozen reference policy. [9]

PPO clip ratio set wrong

With $\epsilon$ too small (less than $0.1$ in the standard $\epsilon$ notation) the clip kicks in on most tokens and learning stalls; with $\epsilon$ too large (above $0.3$ ) the trust-region assumption breaks and training oscillates. The InstructGPT default of $\epsilon = 0.2$ is a reasonable starting point; if the run is unstable, lower $\epsilon$ and check whether KL is exploding before raising it. [9]

SFT data quality dominates everything downstream

RLHF and DPO can only redistribute probability mass over the responses $\pi_{\mathrm{SFT}}$ already places mass on; they cannot teach a behavior the SFT model never demonstrates. Spending time on a tiny but high-quality SFT dataset (a few thousand vetted examples) pays back more than scaling the preference data ten-fold. The cliched ordering (SFT then preference) is engineering reality, not just convention.

Pop quiz

Check yourself

Score: 0 / 5(0 answered)

1. In LoRA, the rank-r update to a frozen weight matrix $W_0 \in \mathbb{R}^{d \times k}$ is parameterized as $\Delta W = (\alpha / r) B A$. The dimensions of $B$ and $A$ are:
2. DPO eliminates the reward model in RLHF by exploiting which property?
3. The KL penalty against pi_ref in the RLHF objective J(theta) = E[r_phi(x, y) - beta * KL(pi_theta || pi_ref)] exists primarily to:
4. In QLoRA, NF4 4-bit quantization is:
5. Prefix tuning and prompt tuning differ from LoRA primarily in:

Interview Q&A

Derive the LoRA rank-r update math and explain why merging back into the base matrix is free at inference.onsite

Setup. Start from a frozen pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$ inside some attention or FFN projection. LoRA freezes $W_0$ and parameterizes the task-specific delta as a low-rank product

\Delta W = \frac{\alpha}{r} B A, \qquad B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k},

with $r \ll \min(d, k)$ . The forward computation becomes $h = (W_0 + \Delta W) x = W_0 x + (\alpha / r) B (A x)$ . At initialization $B = 0$ and $A$ is sampled (Gaussian or Kaiming), so $\Delta W = 0$ and the model starts behaving exactly like the frozen base.

Trainable parameter count. Full fine-tuning trains $d k$ parameters per matrix. LoRA trains $r (d + k)$ . For a representative attention projection at $d = k = 4096$ and $r = 16$ , full is $4096 \times 4096 \approx 1.68 \times 10^7$ parameters; LoRA is $16 \times (4096 + 4096) \approx 1.31 \times 10^5$ parameters. The ratio is $4096 \times 4096 / (16 \times 8192) = 128$ , so LoRA is roughly $128\times$ cheaper per matrix to train and store. Multiply across all attention and FFN projections in the model and the savings carry through end-to-end.

Why merging is free at inference. After training, fold the adapter into the base:

W_{\mathrm{merged}} = W_0 + \frac{\alpha}{r} B A.

$W_{\mathrm{merged}}$ has the same shape as $W_0$ , so the deployed forward path is a single matmul $h = W_{\mathrm{merged}} x$ identical in cost to the original frozen model. There is no extra branch, no extra kernel, no extra parameter at runtime. The adapter exists only at training time; at inference it has been absorbed.

On target modules. The original LoRA paper restricted adaptation to $W_Q$ and $W_V$ in attention because that was sufficient to match full fine-tuning on GLUE and WikiSQL at the time, and it minimized parameter count. Production practice has since moved to LoRA-ing all four attention projections ( $W_Q, W_K, W_V, W_O$ ) and the FFN up- and down-projections, because the marginal cost is small and the marginal lift on instruction-following and reasoning tasks is real. The default in TRL and PEFT today is “all linear layers” [4].

Prove that the DPO loss is equivalent to the optimum of RLHF under a Bradley-Terry preference model.research

Step 1: closed form of the KL-constrained RLHF optimum. The RLHF objective

J(\pi) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot \mid x)}\bigl[r(x, y)\bigr] - \beta \, \mathrm{KL}\bigl(\pi(\cdot \mid x) \,\big\|\, \pi_{\mathrm{ref}}(\cdot \mid x)\bigr)

is a per- $x$ optimization over the simplex of distributions $\pi(\cdot \mid x)$ . Writing the per- $x$ Lagrangian and differentiating with respect to $\pi(y \mid x)$ gives

r(x, y) - \beta \log \frac{\pi(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} - \beta = \mu(x),

where $\mu(x)$ is the multiplier enforcing $\sum_y \pi(y \mid x) = 1$ . Solving for $\pi$ and absorbing the $-\beta$ and $\mu(x)$ into a normalizer yields the well-known closed form

\pi^{*}(y \mid x) = \frac{1}{Z(x)} \, \pi_{\mathrm{ref}}(y \mid x) \exp\!\left(\frac{1}{\beta} r(x, y)\right), \qquad Z(x) = \sum_{y'} \pi_{\mathrm{ref}}(y' \mid x) \exp\!\left(\frac{1}{\beta} r(x, y')\right).

Step 2: invert to express the reward in terms of the optimum. Take the log of the closed form:

r(x, y) = \beta \log \frac{\pi^{*}(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x).

The second term depends only on $x$ , not on $y$ .

Step 3: substitute into Bradley-Terry. The Bradley-Terry preference model gives

P(y_w \succ y_l \mid x) = \sigma\bigl(r(x, y_w) - r(x, y_l)\bigr).

Plug in the inverted reward. The two $\beta \log Z(x)$ terms are identical functions of $x$ alone and cancel exactly in the difference:

P(y_w \succ y_l \mid x) = \sigma\!\left(\beta \log \frac{\pi^{*}(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \beta \log \frac{\pi^{*}(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}\right).

Maximum-likelihood training on the preference dataset $\{(x, y_w, y_l)\}$ , parameterizing $\pi^{*}$ as a learnable policy $\pi_\theta$ , is exactly the DPO loss

\mathcal{L}_{\mathrm{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l)}\!\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}\right)\right].

The reward model has been eliminated because Bradley-Terry only uses the difference of rewards, and the data-dependent partition function $\beta \log Z(x)$ vanishes in that difference.

What is not assumed. Nothing in this proof requires PPO, advantage estimators, or any specific RL convergence result. The equivalence is purely between the optimum of the RLHF objective and the objective of DPO, evaluated against the same Bradley-Terry preference data. Whether a real PPO run actually reaches that optimum is a separate, empirical question [10][9].

Why does PPO use a clipped probability ratio?onsite

The importance-sampling identity. PPO collects a buffer of rollouts under an old policy $\pi_{\mathrm{old}}$ , then performs several gradient steps on $\theta$ before refilling the buffer. The policy-gradient identity rewrites the on-policy expectation as an importance-weighted expectation under $\pi_{\mathrm{old}}$ :

\mathbb{E}_{\pi_\theta}[A(s, a)] = \mathbb{E}_{\pi_{\mathrm{old}}}\!\left[\frac{\pi_\theta(a \mid s)}{\pi_{\mathrm{old}}(a \mid s)} A(s, a)\right].

Define the per-token ratio $\rho_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\mathrm{old}}(a_t \mid s_t)$ . The naive surrogate $L^{\mathrm{IS}}(\theta) = \mathbb{E}[\rho_t A_t]$ is unbiased only when $\pi_\theta$ stays close to $\pi_{\mathrm{old}}$ . If gradient ascent moves $\pi_\theta$ far away during the inner loop, the ratios stop being approximately one, the importance weights become high-variance, and the surrogate is no longer a faithful estimate of the true policy gradient.

The clip. PPO replaces the surrogate with

L^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t\!\left[\min\bigl(\rho_t A_t, \, \mathrm{clip}(\rho_t, 1 - \epsilon, 1 + \epsilon) \, A_t\bigr)\right].

The clip caps how much one update can shift the per-token probability ratio: above $1 + \epsilon$ when $A_t > 0$ , or below $1 - \epsilon$ when $A_t < 0$ , the gradient on that token zeros out. This implements a coarse trust-region without the second-order Hessian-vector products of TRPO.

Tuning $\epsilon$ . The standard value is $\epsilon = 0.2$ . Smaller (e.g., $\epsilon = 0.1$ ) is more conservative: the clip kicks in on more tokens, learning slows. Larger (e.g., $\epsilon = 0.3$ ) is more aggressive: the trust-region assumption breaks and training oscillates. Practical note: if KL is exploding during PPO, lowering $\epsilon$ almost always helps before raising init_kl_coef. The clip controls the per-step move in policy space; the KL coefficient controls drift over many steps. Diagnosing the right knob means looking at where the divergence is happening [9].

Why does RLHF need a reference model and a KL penalty?onsite

The reward model is a learned proxy. The reward model $r_\phi(x, y)$ is fit on a finite preference dataset (tens of thousands of pairs at most). Inside the support of that data it is a reasonable estimator of human preference. Outside the support, it is unreliable in arbitrary directions: there are sequences that score very high under $r_\phi$ but that humans would label as terrible. This is the classic Goodhart situation: an imperfect reward becomes a target and the optimizer finds the gaps.

Unconstrained maximization fails predictably. If the policy maximizes $r_\phi$ without constraint, gradient ascent will discover the sharpest cliffs in $r_\phi$ -space, almost always adversarial sequences that exploit the proxy. The policy collapses onto these mode-seeking outputs and pins probability mass there. Empirically you see reward rising while held-out human-eval quality crashes.

The KL fixes this. Adding $-\beta \, \mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}})$ bounds the policy to a neighborhood of $\pi_{\mathrm{ref}}$ (the SFT model) where $r_\phi$ is still calibrated. In that neighborhood, “score better under $r_\phi$ ” is correlated with “actually preferred by humans”; outside it, the correlation is broken.

A Bayesian reading. The closed-form optimum of the KL-constrained objective is

\pi^{*}(y \mid x) \propto \pi_{\mathrm{ref}}(y \mid x) \, \exp\!\left(\frac{1}{\beta} r_\phi(x, y)\right),

which is exactly the Bayesian posterior of $\pi_{\mathrm{ref}}$ updated under a likelihood proportional to $\exp(r_\phi / \beta)$ . The KL-constrained policy is the prior $\pi_{\mathrm{ref}}$ tilted toward higher reward, not a free maximizer of $r_\phi$ . The prior keeps the geometry sane.

Engineering bottom line. The reference model and the KL are not optional regularizers in some “make the run more stable” sense. They are the only thing that keeps reward maximization from degenerating into the proxy’s failure modes. Every successful RLHF pipeline (InstructGPT, Llama-2-Chat, Claude) carries a frozen reference policy and a non-trivial $\beta$ throughout training [9].

Build it yourself

A weekend-scale project that takes fine-tuning from formulas to a working chat model:

Start from a 7B open-weights base (Llama-2 or Mistral) and a small SFT dataset (a few thousand vetted instruction-response pairs from OpenAssistant or Dolly). Run a LoRA fine-tune at $r = 16$ on all attention and FFN projections; verify the resulting model follows instructions on a few held-out prompts.
Take a public preference dataset (Anthropic HH-RLHF or UltraFeedback). Train a reward model on top of your SFT model: replace the LM head with a scalar regression head, fine-tune on the Bradley-Terry log-likelihood loss. Verify that on held-out preference pairs the reward model picks the preferred response above $70\%$ of the time.
Run the full RLHF loop with TRL’s PPOTrainer: rollouts from the SFT policy, scores from your reward model, PPO updates with init_kl_coef = 0.2 and clip $\epsilon = 0.2$ . Plot reward versus KL during training and confirm KL stays below $10$ nats while reward rises.
Re-do step 3 with TRL’s DPOTrainer on the same preference dataset. Compare wall-clock time and final preference-win-rate on a held-out evaluator. DPO should reach comparable quality in roughly a third of the time.
Stretch: switch the LoRA base to QLoRA NF4. Re-run steps 1 through 4 on a 24 GB consumer GPU and reproduce the QLoRA paper’s claim that quality holds.

Interactive Demo

LoRA Weight Decomposition

LoRA represents weight updates as W' = W + (B × A) where B and A are low-rank matrices.

Rank r = 2

Alpha α = 8

W (8×8)pretrained

B (8×2)LoRA B

A (2×8)LoRA A

W' (8×8)updated

Original params

LoRA params

8r + r8 = 16r

Trainable ratio

50.0%

Reduction

2.0×

fewer params

What to try: Change the rank to see how few parameters LoRA needs to represent weight updates

References

Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019). Parameter-Efficient Transfer Learning for NLP. ICML 2019 (arXiv preprint). arXiv:1902.00751
Li, X. L., and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL 2021 (arXiv preprint). arXiv:2101.00190
Lester, B., Al-Rfou, R., and Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP 2021 (arXiv preprint). arXiv:2104.08691
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022 (arXiv preprint 2021). arXiv:2106.09685
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023 (arXiv preprint). arXiv:2305.14314
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. (2021). Finetuned Language Models Are Zero-Shot Learners. ICLR 2022 (arXiv preprint 2021). arXiv:2109.01652
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022). Scaling Instruction-Finetuned Language Models. arXiv preprint. arXiv:2210.11416
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS 2017 (arXiv preprint). arXiv:1706.03741
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022 (arXiv preprint). arXiv:2203.02155
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 (arXiv preprint). arXiv:2305.18290
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv preprint. arXiv:2212.08073
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org

Scaling Laws & Training Retrieval-Augmented Generation