Offline RL & RLHF

Background

Offline RL (batch RL) trains from a fixed dataset of past interactions, without live environment access
RLHF (Christiano et al. 2017) uses human preference labels to train reward models that guide RL fine-tuning of LLMs
InstructGPT (2022) and subsequent ChatGPT showed RLHF is the key to instruction-following LLMs

The Concept

Offline RL trains a policy from a fixed historical dataset without further environment interaction, enabling RL on data collected by other agents. RLHF applies offline RL to LLMs: human preferences train a reward model, which then guides PPO fine-tuning of the language model.

How it Links

Connects directly to fine-tuning and PEFT - RLHF is the fine-tuning method that creates chat-aligned LLMs
Constitutional AI (Anthropic) and DPO (Direct Preference Optimization) are alternatives to the full RLHF pipeline
Dataset shift and distributional issues in offline RL connect to data leakage and covariate shift concepts

Concrete Example

ChatGPT alignment: base GPT-3 sometimes outputs harmful or useless text. RLHF fix: (1) humans rank GPT outputs for helpfulness/safety, (2) train a reward model on these rankings, (3) PPO fine-tune GPT to maximize reward model score. The result follows instructions reliably - not because the architecture changed, but because the reward signal changed.

Prerequisites

Foundations of RL, Fine-Tuning & PEFT, Transformers

Learning outcomes

Explain the offline RL problem: why distribution shift and extrapolation error make off-policy learning from a static dataset fundamentally harder than online RL.
Describe how Conservative Q-Learning (CQL) penalises out-of-distribution actions and how IQL avoids querying OOD actions entirely.
Explain the Decision Transformer framing: RL as sequence modeling conditioned on desired return.
Derive the DPO objective from the KL-regularised RLHF objective, showing how the partition function cancels and why no separate reward model is needed.
Explain Constitutional AI and RLAIF as mechanisms for scalable alignment that replace human labellers with model-generated feedback.

TL;DR

Offline RL learns from a fixed dataset without environment interaction, but distribution shift causes Q-values to extrapolate dangerously for unseen actions — CQL and IQL constrain this conservatively. RLHF trains a reward model from human preferences and fine-tunes with PPO; DPO eliminates the separate reward model by directly expressing the RLHF objective as a classification loss over the policy itself, which is why it is simpler and more stable.

Intuition

Standard RL assumes the agent can explore: try an action, observe the outcome, update. Offline RL removes this privilege — you have a dataset of $(s, a, r, s')$ tuples collected by some behaviour policy, and the goal is to extract the best possible policy from that fixed data. This sounds like supervised learning, but the distribution shift problem makes it far harder: a learned Q-function may assign high values to (state, action) pairs that look good on paper but were never tried — and acting on those inflated values during deployment leads to catastrophic failures.

Insight

The core offline RL problem: the Q-function is only accurate near the support of the dataset. Any action outside that support can be assigned arbitrarily high Q-values by function approximation, and greedy policies will exploit these hallucinated values.

The Offline RL Problem: Distribution Shift and Extrapolation Error

Let the behaviour policy that collected the dataset be $\mu(a \mid s)$ and the target policy be $\pi(a \mid s)$ . The Bellman backup includes a max over actions:

Q(s, a) \leftarrow r + \gamma \max_{a'} Q(s', a')

When $\pi$ selects actions outside $\text{supp}(\mu)$ , the TD target bootstraps from $Q$ values never trained on real data. Bellman residual minimisation will try to reduce error in regions where data exist — but the max operation systematically selects the worst offenders (highest but least accurate $Q$ values). Levine et al. [9] call this extrapolation error.

Conservative Q-Learning (CQL)

Kumar et al. (2020) [1] propose adding a penalty to the standard Bellman objective that lowers $Q$ values for out-of-distribution (OOD) actions and raises them for in-distribution actions:

\min_Q \; \alpha \left( \mathbb{E}_{s \sim \mathcal{D}}\left[\log \sum_a \exp Q(s,a)\right] - \mathbb{E}_{(s,a) \sim \mathcal{D}}[Q(s,a)] \right) + \frac{1}{2} \mathbb{E}_{(s,a,s') \sim \mathcal{D}}\left[(Q(s,a) - \hat{\mathcal{B}}^\pi Q(s,a))^2\right]

The first term is a soft maximum that assigns high values globally to all actions, then subtracts back the expectation under the data distribution — the net effect is to push $Q$ below the data values for unseen $(s, a)$ pairs. The hyperparameter $\alpha$ controls conservatism. CQL provides theoretical guarantees that the learned $Q$ is a lower bound on the true $Q^\pi$ , making the greedy policy pessimistically safe.

Implicit Q-Learning (IQL)

Kostrikov et al. (2021) [2] avoid the problem of ever querying OOD actions by reformulating the Bellman update to use only in-dataset transitions. The key trick is expectile regression for the value function:

\mathcal{L}_V(V) = \mathbb{E}_{(s,a) \sim \mathcal{D}} \left[ L_\tau^\text{expectile}(Q^{\hat{\mu}}(s,a) - V(s)) \right]

where $L_\tau^\text{expectile}(u) = |\tau - \mathbb{1}[u < 0]| \cdot u^2$ and $\tau \in (0.5, 1)$ is a quantile parameter. With $\tau$ close to 1, $V(s)$ approximates the maximum in-dataset Q-value without ever evaluating the policy at novel states. Policy extraction then uses advantage-weighted regression:

\pi = \arg\max_\pi \mathbb{E}_{(s,a) \sim \mathcal{D}} \left[ \exp(\beta (Q(s,a) - V(s))) \log \pi(a \mid s) \right]

IQL achieves state-of-the-art on D4RL benchmarks and is simpler to tune than CQL.

Decision Transformer: RL as Sequence Modeling

Chen et al. (2021) [3] reframe offline RL entirely: instead of learning a Q-function, train a causal Transformer to predict actions conditioned on past returns, states, and actions. At test time, condition on a desired high return and let the model produce the actions:

a_t \sim \pi_\theta(a_t \mid \hat{R}_t, s_t, a_{t-1}, R_{t-1}, s_{t-1}, \ldots)

where $\hat{R}_t$ is the “return-to-go” (desired future return, decremented after each step). This approach avoids Bellman backups entirely, treating the whole trajectory as a language-model-style prediction problem. It matches or exceeds CQL on several D4RL tasks and scales gracefully with model size, making it a precursor to the trajectory-level reasoning seen in LLM-based agents.

RLHF: The Three-Stage Pipeline

Reinforcement Learning from Human Feedback [4] [7] aligns language models through three stages:

Stage 1 — Supervised Fine-Tuning (SFT): Fine-tune the base LLM on high-quality human-written demonstrations to produce $\pi_{\text{SFT}}$ .

Stage 2 — Reward Modeling (RM): Collect pairs of completions $(y_w, y_l)$ for the same prompt $x$ with human labels indicating which is preferred. Train a scalar reward model $r_\phi(x, y)$ via the Bradley-Terry log-likelihood:

\mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)) \right]

Stage 3 — PPO Fine-Tuning: Maximise the reward model signal while penalising KL divergence from the SFT reference policy to prevent reward hacking:

\max_\pi \; \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi}[r_\phi(x,y)] - \beta \cdot \mathbb{KL}[\pi(y \mid x) \| \pi_{\text{SFT}}(y \mid x)]

Warning

Reward hacking: the policy may learn to game the reward model — for example, producing extremely long outputs if the RM spuriously scores length highly, or generating confident-sounding nonsense. The KL penalty $\beta$ limits how far the policy can drift, but tuning it is critical.

DPO: Direct Preference Optimization

Rafailov et al. (2023) [5] show that the RLHF objective can be solved in closed form, eliminating the need for a separate reward model and PPO training loop.

Deriving the DPO objective from the RLHF KL-regularised objective

Step 1 — The KL-regularised RLHF objective.

\max_\pi \; \mathbb{E}_{x \sim \mathcal{D}} \left[ \mathbb{E}_{y \sim \pi}[r(x,y)] - \beta \, \mathbb{KL}[\pi(y|x) \| \pi_{\text{ref}}(y|x)] \right]

Step 2 — Optimal policy in closed form.

This is a classic KL-regularised optimisation. The functional derivative with respect to $\pi(y|x)$ yields the optimal policy:

\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\!\left(\frac{r(x,y)}{\beta}\right)

where $Z(x) = \sum_y \pi_{\text{ref}}(y|x) \exp(r(x,y)/\beta)$ is the (intractable) partition function.

Step 3 — Invert to express reward in terms of optimal policy.

Taking the log and rearranging:

r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)

Step 4 — Substitute into the Bradley-Terry preference model.

The Bradley-Terry probability that $y_w$ is preferred to $y_l$ is:

p(y_w \succ y_l \mid x) = \sigma(r(x, y_w) - r(x, y_l))

Substituting the expression for $r$ and noting that $\log Z(x)$ appears in both terms and cancels:

p(y_w \succ y_l \mid x) = \sigma\!\left(\beta \log \frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)

Step 5 — The DPO training loss.

Replacing $\pi^*$ with the trainable policy $\pi_\theta$ and taking the negative log-likelihood:

\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right) \right]

This is a binary cross-entropy loss over log-likelihood ratios of the trainable policy relative to the reference policy. No reward model, no PPO, no RL training loop. The policy $\pi_\theta$ is simultaneously the language model and the implicit reward model.

Constitutional AI and RLAIF

Constitutional AI (CAI) [6] (Anthropic, 2022) scales alignment by replacing expensive human preference labels with model-generated feedback:

Generate responses with the SFT model (including some harmful ones).
Ask the model to critique each response against a set of principles (the “constitution”), then revise it.
Use the original vs revised pairs to train a Preference Model (PM) — the equivalent of the RLHF reward model, but trained on AI-generated labels.
Fine-tune with RL (RL-CAI) or with DPO against the AI-labeled preferences (RLAIF).

Insight

RLAIF is the generalisation: any pipeline where the preference signal comes from a model rather than humans. Constitutional AI is Anthropic’s specific instantiation with an explicit constitution of principles.

Method Comparison

Method	Data needed	Requires RM?	Online/Offline	Compute	Stability	Reward hacking risk
SFT only	Demonstrations	No	Offline	Low	High	None
RLHF (PPO)	Preferences + demonstrations	Yes (explicit)	Online	High (4 models)	Low (PPO instability)	High if KL not tuned
DPO	Preferences	No (implicit in pi)	Offline	Medium (SFT-level)	High	Lower (no separate RM)
IPO (Azar 2023)	Preferences	No	Offline	Medium	High	Lower (direct preference target)
KTO (Ethayarajh 2024)	Single-label (good/bad)	No	Offline	Medium	High	Lower
Constitutional AI / RLAIF	AI-generated preferences	Yes (AI-labeled)	Either	Medium	Medium	Medium

Real-World: ChatGPT, Claude, Llama 2

ChatGPT (InstructGPT) [7]: classic three-stage RLHF with PPO. Human labellers rank completions; RM trained; PPO optimises with KL penalty.
Claude [6]: Constitutional AI reduces reliance on human preference labels. The constitution specifies principles; the model self-critiques and revises.
Llama 2-Chat [10]: two reward models (helpfulness and safety) trained separately; PPO optimises a linear combination with a margin-based reward.
DPO in practice: many open-source fine-tunes (Zephyr, OpenHermes) use DPO because it requires no separate RM training and is as simple as running SFT on preference-weighted data.

Warning

Length bias in preference datasets: humans often prefer longer outputs even when the content is equivalent. This causes reward models to spuriously score length, and DPO to increase output length. Explicitly controlling length in the preference collection phase or adding length penalties helps.

Pop Quiz

Check yourself

Score: 0 / 5(0 answered)

1. What is the primary challenge that makes offline RL harder than online RL?
2. How does CQL prevent extrapolation error?
3. In DPO, what mathematical quantity cancels that allows the derivation to work?
4. What does RLAIF replace compared to standard RLHF?
5. The Decision Transformer frames offline RL as:

Interview Q&A

Derive the DPO objective from the RLHF objective, explaining each step.staff

The full derivation is a 5-step argument:

Start from the KL-regularised RLHF objective: $\max_\pi \mathbb{E}[r(x,y)] - \beta \, \mathbb{KL}[\pi \| \pi_{\text{ref}}]$ .
Solve the functional optimisation via calculus of variations. Setting the derivative to zero gives $\pi^*(y|x) \propto \pi_{\text{ref}}(y|x) \exp(r(x,y)/\beta)$ .
Invert this to express the reward: $r(x,y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$ .
Substitute into the Bradley-Terry preference model $p(y_w \succ y_l) = \sigma(r(y_w) - r(y_l))$ . The $\log Z(x)$ terms cancel because they appear in both $r(y_w)$ and $r(y_l)$ .
Replace $\pi^*$ with the trainable $\pi_\theta$ and take the negative log-likelihood to get the DPO cross-entropy loss over log-ratio differences.

The result is a supervised binary classification loss requiring no reward model and no RL. DPO works because the optimal RLHF policy already implicitly encodes a reward model; DPO just directly optimises that implicit encoding.

Why is offline RL harder than online RL? What is extrapolation error and how does CQL address it?onsite

In online RL, every action the policy takes generates a new $(s, a, r, s')$ tuple, so the Q-function is trained on its own distribution. In offline RL, the dataset was generated by a fixed behaviour policy $\mu$ . When we run policy improvement (take the max-Q action), we select actions that may be outside the support of $\mu$ . The Q-function was never trained on those $(s, a)$ pairs, so function approximation can assign them arbitrarily high values — a phenomenon called extrapolation error. The policy then exploits these spurious maxima and fails in deployment.

CQL addresses this by adding a pessimistic regulariser to the Bellman objective: a term that simultaneously maximises the log-sum-exp of Q over all actions (encourages high values globally) and subtracts the in-data expectation (anchors values down in regions where data exist). The net effect is that OOD actions have lower Q-values than in-distribution actions, making the greedy policy pessimistically conservative.

What is reward hacking, and how does the KL penalty in RLHF mitigate it?phone

Reward hacking is Goodhart’s Law applied to RL: “When a measure becomes a target, it ceases to be a good measure.” The reward model $r_\phi$ is a neural network trained on a finite preference dataset; it approximates human preferences but is not identical to them. A powerful policy optimised against $r_\phi$ can find inputs that score highly on the learned reward but are actually poor outputs — for example, extremely long responses, repetitive confident-sounding text, or adversarial patterns that exploit reward-model weaknesses.

The KL penalty $\beta \, \mathbb{KL}[\pi \| \pi_{\text{SFT}}]$ prevents the policy from drifting too far from the SFT reference. A large $\beta$ keeps the policy conservative (close to SFT, less likely to hack), while a small $\beta$ allows more reward-driven optimisation. In practice $\beta$ is treated as a hyperparameter that trades off alignment-to-reward-model vs divergence-from-human-intent.

References

Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS 2020 (arXiv preprint). arXiv:2006.04779
Kostrikov, I., Nair, A., and Levine, S. (2021). Offline Reinforcement Learning with Implicit Q-Learning. ICLR 2022 (arXiv preprint 2021). arXiv:2110.06169
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. (2021). Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS 2021 (arXiv preprint). arXiv:2106.01345
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS 2017 (arXiv preprint). arXiv:1706.03741
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023 (arXiv preprint). arXiv:2305.18290
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv preprint. arXiv:2212.08073
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022 (arXiv preprint). arXiv:2203.02155
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv preprint. arXiv:1707.06347
Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. arXiv preprint. arXiv:2005.01643
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint. arXiv:2307.09288

Model-Based RL & MuZero Multi-Agent RL