Diffusion Models

Background

Ho et al. (2020) introduced DDPM (Denoising Diffusion Probabilistic Models) as a new class of generative models
Stable Diffusion (2022) made high-quality image generation accessible by moving diffusion into latent space
Now applied to audio (AudioLDM), video (Sora), proteins (RFdiffusion), and molecular design

The Concept

Diffusion models learn to reverse a gradual noising process. During training, Gaussian noise is added to data step by step. The model (a U-Net or DiT) learns to predict and remove the noise at each step. At generation, pure noise is denoised iteratively to produce realistic samples.

How it Links

Stable Diffusion combines a VAE encoder/decoder with a latent diffusion model - connects to autoencoders
Score matching and the diffusion process connect to stochastic differential equations (SDEs)
Classifier-free guidance connects to conditioning and CLIP-based text-to-image generation

Concrete Example

Text-to-image: "a red panda coding on a laptop". The model starts with random noise, then takes 50 denoising steps guided by the CLIP embedding of the text. Each step removes a little noise while steering toward the text description. The final image emerges from pure noise, guided by language.

Prerequisites

Autoencoders & VAEs, Generative Adversarial Networks, The Transformer Architecture, Backpropagation & Optimizers

Learning outcomes

Write the DDPM forward process and derive the closed-form one-shot noising q(x_t | x_0).
Derive the DDPM variational lower bound and reduce it to the simple epsilon-prediction MSE objective.
Connect noise prediction to the score nabla_x log p_t(x) via Tweedie's formula and state the SDE / probability-flow ODE picture.
Explain DDIM deterministic sampling and why it lets you skip from 1000 to 50 steps with the same trained network.
Derive classifier-free guidance and explain how the guidance weight w trades sample fidelity against diversity.
Compare diffusion to GANs, VAEs, and autoregressive image models on training stability, mode coverage, sample quality, and sampling cost.

TL;DR

A diffusion model defines a fixed forward Markov chain that gradually corrupts data with Gaussian noise, then learns the reverse chain by minimising a variational lower bound. Ho 2020 [2] showed the bound collapses to a simple MSE objective: predict the noise $\varepsilon$ that was added to $x_0$ to get $x_t$ . The trained network $\varepsilon_\theta(x_t, t)$ is equivalent to a score model $\nabla_x \log p_t(x)$ up to a known scaling [3][4], which lets you sample with stochastic Langevin updates, deterministic DDIM updates [5], or any solver for the underlying SDE. Classifier-free guidance [6] trains conditional and unconditional networks jointly and at sampling time extrapolates the conditional prediction to sharpen text-following at the cost of diversity. Latent diffusion [7] runs the diffusion in a VAE latent space rather than at the pixel level and is the architecture behind Stable Diffusion and most production image generators. The standard backbones are a U-Net [8] with skip connections or, at scale, a DiT [9] built on Transformer blocks. Diffusion replaced GANs as the SOTA image generator because training is stable, mode coverage is high, and conditioning is easy.

Intuition

Two pictures, both correct. The first is corruption and denoising. Take a clean image $x_0$ and add a tiny bit of Gaussian noise. Repeat for $T$ steps until you are left with a sample that is statistically indistinguishable from pure noise $x_T \sim \mathcal{N}(0, I)$ . If you can train a network that, given a slightly-noisier $x_t$ , predicts the slightly-cleaner $x_{t-1}$ , then sampling is just running that network $T$ times starting from pure noise.

The second is score matching. At each noise level $t$ the data has a smoothed density $p_t(x)$ . The score is the gradient of its log-density, $\nabla_x \log p_t(x)$ , which is a vector field that points “uphill” toward higher density. If a network can learn this field for every noise level, you can sample by following Langevin dynamics: start from noise, take small steps in the score direction with a bit of fresh noise, and end up at a sample of $p_0 = p_{\mathrm{data}}$ . The miracle is that noise prediction and score estimation are the same network up to a known scaling [3][4]; the choice between MSE on $\varepsilon$ and score matching is a parameterisation choice, not a modelling choice.

This unification is why diffusion replaced GANs [10]:

Training is stable. GAN training is a saddle-point optimisation on a non-convex non-concave objective; diffusion training is plain MSE, single network, no adversary. Loss curves go down monotonically.
Mode coverage is high. GANs collapse onto subsets of the data distribution; diffusion is trained to model the entire data density, so it covers modes by construction.
Conditioning is easy. A diffusion model conditions on text, class labels, or image structure simply by passing the conditioner as an extra input to $\varepsilon_\theta$ . Classifier-free guidance turns the conditioning strength into a runtime knob.
Sampling is expensive. The price is that you call the network many times per sample. The whole DDIM / consistency-model / distillation literature is about reducing this from 1000 steps to 50, then 10, then 1.

The training objective looks too simple

The DDPM training loop is loss = (eps - eps_theta(noisy_x, t)).pow(2).mean(). Five lines. There is no adversarial loss, no KL regulariser, no auxiliary head, no temperature, no replay buffer. Every modern image generator is a small convolutional or Transformer network trained with that one line of code, and the entire literature since 2020 has been finding ways to use that one line better. This is the kind of architectural simplicity that wins long-term.

Timeline

2015Sohl-Dickstein et al. — diffusion as nonequilibrium thermodynamicsfoundational
First proposal to learn data distributions via a forward noising process and a learned reverse process; established the variational lower bound but used difficult-to-train transition kernels.
2019Song & Ermon — score-based generative modeling (NCSN)score
Score matching with a noise-conditional network, sampled by annealed Langevin dynamics. The "score" view that DDPM later turned out to be equivalent to.
2020Ho et al. — DDPMbreakthrough
Showed the variational bound reduces to a simple MSE on noise prediction, made diffusion competitive with GANs on FID, and gave the recipe (U-Net, sinusoidal time embedding, eps-parameterisation) the field still uses.
2020Song et al. — SDE / probability-flow ODEunification
Unified DDPM and NCSN as discretisations of forward and reverse stochastic differential equations; introduced the deterministic ODE sampler.
2020Song, Meng, Ermon — DDIMsampling
Deterministic non-Markovian sampling chain that reuses the same trained eps_theta and reaches comparable quality with 50 steps instead of 1000.
2021Ho & Salimans — classifier-free guidanceconditioning
Joint training of conditional and unconditional eps_theta; at sampling, extrapolate to sharpen conditioning. Replaced classifier guidance and is the standard knob in every text-to-image system.
2021Rombach et al. — Latent Diffusion / Stable Diffusionefficiency
Run diffusion in a VAE latent space rather than pixels. ~64x cheaper for the same image resolution. Is the architecture behind Stable Diffusion 1/2/XL.
2022Peebles & Xie — DiTarchitecture
Replaces the U-Net with a plain Transformer (with adaptive layer norm conditioning on time and class). Better scaling, used in Sora and Stable Diffusion 3.

DDPM forward and reverse processes

The forward Markov chain and its closed-form marginal q(x_t | x_0)

The DDPM forward process is a fixed (no learned parameters) Markov chain that gradually adds Gaussian noise according to a variance schedule $\beta_1, \ldots, \beta_T \in (0, 1)$ :

q(x_t \mid x_{t-1}) = \mathcal{N}\bigl(x_t;\ \sqrt{1 - \beta_t}\, x_{t-1},\ \beta_t I\bigr).

Define $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s=1}^{t} \alpha_s$ . The schedule is chosen so $\bar\alpha_T \approx 0$ , i.e. $x_T$ is essentially pure noise.

Closed-form one-shot noising. A central convenience is that the marginal $q(x_t \mid x_0)$ has a closed form. Using the reparameterisation $x_t = \sqrt{\alpha_t}\, x_{t-1} + \sqrt{\beta_t}\, z_t$ with $z_t \sim \mathcal{N}(0, I)$ and unrolling:

\begin{aligned} x_t &= \sqrt{\alpha_t}\, x_{t-1} + \sqrt{1 - \alpha_t}\, z_t \\ &= \sqrt{\alpha_t \alpha_{t-1}}\, x_{t-2} + \sqrt{\alpha_t (1 - \alpha_{t-1})}\, z_{t-1} + \sqrt{1 - \alpha_t}\, z_t \\ &\;\vdots \\ &= \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon \end{aligned}

where $\varepsilon$ is the aggregate of all the per-step noises. Sums of independent zero-mean Gaussians are Gaussian, and a calculation of variance shows the aggregate is unit-variance, so $\varepsilon \sim \mathcal{N}(0, I)$ and

\boxed{\ q(x_t \mid x_0) = \mathcal{N}\bigl(x_t;\ \sqrt{\bar\alpha_t}\, x_0,\ (1 - \bar\alpha_t) I\bigr).\ }

This is the equation that makes everything else possible: training does not need to simulate the chain step by step. To produce a training pair $(x_t, t, \varepsilon)$ from $x_0$ , you sample $t$ uniformly from $\{1, \ldots, T\}$ , sample $\varepsilon \sim \mathcal{N}(0, I)$ , and compute $x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon$ in $O(d)$ time.

Posterior $q(x_{t-1} \mid x_t, x_0)$ . This conditional is also Gaussian and is the target of the learned reverse step. By Bayes’ rule on Gaussians:

q(x_{t-1} \mid x_t, x_0) = \mathcal{N}\bigl(x_{t-1};\ \tilde\mu_t(x_t, x_0),\ \tilde\beta_t I\bigr),

with

\tilde\mu_t(x_t, x_0) = \frac{\sqrt{\bar\alpha_{t-1}}\, \beta_t}{1 - \bar\alpha_t}\, x_0 + \frac{\sqrt{\alpha_t}\, (1 - \bar\alpha_{t-1})}{1 - \bar\alpha_t}\, x_t, \qquad \tilde\beta_t = \frac{1 - \bar\alpha_{t-1}}{1 - \bar\alpha_t}\, \beta_t.

The mean is a deterministic linear combination of $x_0$ and $x_t$ ; the variance is a known scalar. This is what the reverse network has to match.

Reverse model. The learned reverse process is also Gaussian:

p_\theta(x_{t-1} \mid x_t) = \mathcal{N}\bigl(x_{t-1};\ \mu_\theta(x_t, t),\ \Sigma_\theta(x_t, t)\bigr).

Ho 2020 fixes $\Sigma_\theta = \sigma_t^2 I$ to either $\beta_t I$ or $\tilde\beta_t I$ (both work, the latter is slightly tighter) and only trains $\mu_\theta$ . Sampling starts at $x_T \sim \mathcal{N}(0, I)$ and applies $T$ ancestral steps $x_{t-1} = \mu_\theta(x_t, t) + \sigma_t z$ with $z \sim \mathcal{N}(0, I)$ .

DDPM ELBO and the epsilon-prediction objective

This is the centerpiece. The simplification from a multi-term variational bound to a single-line MSE is the result that made diffusion practical.

Full derivation of the simple MSE training objective from the variational lower bound

Step 1: variational upper bound on the negative log-likelihood. Like a hierarchical VAE, treat $x_{1:T}$ as latents. By Jensen’s inequality applied to the convex function $-\log$ :

-\log p_\theta(x_0) = -\log \int p_\theta(x_{0:T})\, dx_{1:T} \le \mathbb{E}_{q(x_{1:T} \mid x_0)} \!\left[-\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)}\right] =: L_{\mathrm{vlb}}.

We minimise this upper bound; if it is tight, we are minimising the negative log-likelihood directly.

Step 2: factor both Markov chains. The forward chain factorises as $q(x_{1:T} \mid x_0) = \prod_{t=1}^{T} q(x_t \mid x_{t-1})$ and the reverse as $p_\theta(x_{0:T}) = p(x_T)\, \prod_{t=1}^{T} p_\theta(x_{t-1} \mid x_t)$ . The bound becomes

L_{\mathrm{vlb}} = \mathbb{E}_q\!\left[ -\log p(x_T) - \sum_{t=1}^{T} \log \frac{p_\theta(x_{t-1} \mid x_t)}{q(x_t \mid x_{t-1})} \right].

Step 3: rewrite each forward transition with Bayes’ rule. Use $q(x_t \mid x_{t-1}) = q(x_t \mid x_{t-1}, x_0)$ (the chain is Markov, $x_0$ is redundant) and apply Bayes’ rule to flip the conditioning:

q(x_t \mid x_{t-1}, x_0) = \frac{q(x_{t-1} \mid x_t, x_0)\, q(x_t \mid x_0)}{q(x_{t-1} \mid x_0)}.

Plugging into the bound and telescoping the $q(x_t \mid x_0) / q(x_{t-1} \mid x_0)$ factors over $t$ collapses everything except the boundary terms. After the dust settles (Sohl-Dickstein 2015, Ho 2020 Appendix A), the bound regroups into

L_{\mathrm{vlb}} = \underbrace{\mathbb{E}_q\!\left[ D_{\mathrm{KL}}\bigl(q(x_T \mid x_0)\,\|\,p(x_T)\bigr)\right]}_{L_T} + \sum_{t=2}^{T} \underbrace{\mathbb{E}_q\!\left[ D_{\mathrm{KL}}\bigl(q(x_{t-1} \mid x_t, x_0)\,\|\,p_\theta(x_{t-1} \mid x_t)\bigr)\right]}_{L_{t-1}} + \underbrace{\mathbb{E}_q\!\left[-\log p_\theta(x_0 \mid x_1)\right]}_{L_0}.

Step 4: identify which terms have parameters.

$L_T$ has no learnable parameters. The forward chain is fixed and $p(x_T) = \mathcal{N}(0, I)$ , so $L_T$ is a constant determined by the schedule. Drop it from the loss.
$L_0$ is a discrete-pixel decoder term that Ho 2020 handles separately. For the main argument we ignore it; in code it is a small log-likelihood term that goes to zero quickly.
The bulk of the loss is the sum of $L_{t-1}$ for $t = 2, \ldots, T$ . Each is a KL between two Gaussians with the same variance $\sigma_t^2$ (both forward posterior and reverse model use a fixed isotropic variance), so the KL collapses to a squared distance between means:

L_{t-1} = \mathbb{E}_q\!\left[ \frac{1}{2 \sigma_t^2} \,\bigl\lVert \tilde\mu_t(x_t, x_0) - \mu_\theta(x_t, t) \bigr\rVert^2 \right] + C,

where $C$ is a constant that does not depend on $\theta$ .

Step 5: reparameterise. Plug in $x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\, \varepsilon) / \sqrt{\bar\alpha_t}$ from the closed-form one-shot noising. Substituting into $\tilde\mu_t(x_t, x_0)$ and simplifying gives

\tilde\mu_t(x_t, x_0) = \frac{1}{\sqrt{\alpha_t}} \!\left( x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\, \varepsilon \right).

This suggests parameterising the reverse mean in the same form, with a learned function $\varepsilon_\theta(x_t, t)$ in place of $\varepsilon$ :

\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}} \!\left( x_t - \frac{\beta_t}{\sqrt{1 - \bar\alpha_t}}\, \varepsilon_\theta(x_t, t) \right).

Substitute both expressions into the squared-distance form of $L_{t-1}$ . The $x_t$ terms cancel:

L_{t-1} = \mathbb{E}_q\!\left[ \frac{\beta_t^2}{2 \sigma_t^2 \alpha_t (1 - \bar\alpha_t)} \, \bigl\lVert \varepsilon - \varepsilon_\theta(x_t, t) \bigr\rVert^2 \right].

Step 6: drop the per- $t$ weighting. Ho 2020’s empirical observation: training works at least as well, often better, if you replace the $\beta_t^2 / (2 \sigma_t^2 \alpha_t (1 - \bar\alpha_t))$ weight with $1$ . The simplified objective is

\boxed{\ L_{\mathrm{simple}}(\theta) = \mathbb{E}_{t \sim \mathcal{U}\{1, T\},\ x_0 \sim p_{\mathrm{data}},\ \varepsilon \sim \mathcal{N}(0, I)} \,\left[ \bigl\lVert \varepsilon - \varepsilon_\theta\bigl(\sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon,\ t\bigr)\bigr\rVert^2 \right].\ }

In code:

def loss(model, x0, beta_schedule):
    t = torch.randint(0, T, (x0.size(0),), device=x0.device)
    alpha_bar = compute_alpha_bar(beta_schedule, t)        # (B, 1, 1, 1)
    eps = torch.randn_like(x0)
    x_t = alpha_bar.sqrt() * x0 + (1 - alpha_bar).sqrt() * eps
    eps_pred = model(x_t, t)
    return F.mse_loss(eps_pred, eps)

That is the entire training loop. No adversarial loss, no KL term in the loss, no auxiliary head, no temperature, no schedule on the loss. Every line of complexity in DDPM lives in (a) the variance schedule, (b) the network architecture, and (c) the sampler. The objective is plain MSE.

Why the simple loss is fine, intuitively. The dropped weight de-emphasises tiny $t$ (almost-clean images, where the network’s job is easy and the loss is tiny anyway) and emphasises mid-range $t$ where most of the perceptually important denoising happens. Empirically this is the right inductive bias for image quality even though it loosens the variational bound on $\log p(x_0)$ .

Always verify the variance schedule and t-indexing

Two off-by-one bugs are responsible for most “diffusion is not training” tickets. First, indexing: some implementations sample $t \in \{0, \ldots, T-1\}$ and others $\{1, \ldots, T\}$ ; the $\bar\alpha_t$ vector has to be indexed consistently with whichever convention $\varepsilon_\theta$ expects. Second, $\bar\alpha_0$ should be $1$ (no noise yet); if you accidentally compute $\bar\alpha$ as a length- $T$ cumulative product without the prepended $1$ , your loss-weighting is silently shifted. Always plot the empirical signal-to-noise ratio $\bar\alpha_t / (1 - \bar\alpha_t)$ across $t$ before training and confirm it goes from $\gg 1$ to $\ll 1$ .

The simple loss is not the variational bound

L_simple drops the $\beta_t^2 / (2 \sigma_t^2 \alpha_t (1 - \bar\alpha_t))$ per-timestep weight and is therefore not a valid lower bound on $\log p(x_0)$ . If you care about likelihood (e.g. for compression or for anomaly detection thresholds) you must either keep the variational weighting or use the v-prediction parameterisation (Salimans 2022) which is more stable. For pure sample-quality work, $L_{\mathrm{simple}}$ is the right loss; for log-likelihood claims it is not.

Score-based view and Tweedie’s formula

Why epsilon-prediction is score-matching with a known scaling

Tweedie’s formula. If $y = x + \sigma z$ with $z \sim \mathcal{N}(0, I)$ and $x \sim p(x)$ , then the posterior mean of $x$ given $y$ is

\mathbb{E}[x \mid y] = y + \sigma^2 \nabla_y \log p_\sigma(y),

where $p_\sigma$ is the marginal density of $y$ (a.k.a. the convolution of $p$ with $\mathcal{N}(0, \sigma^2 I)$ ). This is a classical empirical-Bayes identity (Robbins, then Tweedie). It says: knowing the score of the noisy density is equivalent to knowing the optimal denoiser.

Apply it to DDPM. Conditioned on $x_0$ , the marginal $q(x_t \mid x_0) = \mathcal{N}(\sqrt{\bar\alpha_t}\, x_0, (1 - \bar\alpha_t) I)$ . The unconditional marginal $q(x_t)$ is the convolution of the data with this Gaussian. Tweedie gives

\mathbb{E}[\sqrt{\bar\alpha_t}\, x_0 \mid x_t] = x_t + (1 - \bar\alpha_t) \nabla_{x_t} \log q(x_t).

But from the closed-form noising, $\sqrt{\bar\alpha_t}\, x_0 = x_t - \sqrt{1 - \bar\alpha_t}\, \varepsilon$ , and therefore

\mathbb{E}[\varepsilon \mid x_t] = -\sqrt{1 - \bar\alpha_t}\, \nabla_{x_t} \log q(x_t).

The optimal $\varepsilon_\theta(x_t, t)$ is exactly the (negative, scaled) score of the noisy data density. Training $\varepsilon_\theta$ with MSE is therefore denoising score matching in disguise: the same network, the same loss, two parameterisations.

Why this matters. Once you accept the score view, all of generative modelling at noise level $t$ is just running a Langevin chain on $p_t$ , and DDPM’s ancestral sampler is one specific discretisation of this chain. Other discretisations give you DDIM (deterministic), the probability-flow ODE (deterministic continuous-time), or any black-box ODE solver. The same trained network powers all of them. That is why a model trained with DDPM at $T = 1000$ can be sampled at 50 steps with DDIM, or 10 steps with a higher-order solver, or 1 step with consistency distillation.

SDE view, briefly. Song et al. 2020 [4] let $T \to \infty$ and write the forward chain as the SDE $dx = f(x, t)\, dt + g(t)\, dw$ . The reverse-time SDE is $dx = [f(x, t) - g(t)^2 \nabla_x \log p_t(x)]\, dt + g(t)\, d\bar w$ , i.e. forward dynamics minus a score-driven drift. The deterministic probability-flow ODE has the same marginals as the SDE but no Brownian term: $dx = [f(x, t) - \tfrac{1}{2} g(t)^2 \nabla_x \log p_t(x)]\, dt$ . DDIM is the discretisation of this ODE for the variance-preserving schedule.

DDIM: deterministic non-Markovian sampling

The DDPM ancestral sampler needs $T = 1000$ steps to produce a sample of comparable quality to its training schedule. DDIM [5] is a clever observation: the joint distribution $q(x_{1:T} \mid x_0)$ is not unique given the marginals $q(x_t \mid x_0)$ . Song, Meng, and Ermon construct a non-Markovian forward chain with the same one-shot marginals but a different conditional structure. Crucially, the variational bound for this new chain reduces to the same simple noise-prediction objective, so any $\varepsilon_\theta$ trained the DDPM way is also a valid DDIM model.

DDIM’s reverse step is

x_{t-1} = \sqrt{\bar\alpha_{t-1}}\, \hat x_0(x_t, t) + \sqrt{1 - \bar\alpha_{t-1} - \sigma_t^2}\, \varepsilon_\theta(x_t, t) + \sigma_t z, \qquad z \sim \mathcal{N}(0, I),

where $\hat x_0(x_t, t) = (x_t - \sqrt{1 - \bar\alpha_t}\, \varepsilon_\theta(x_t, t)) / \sqrt{\bar\alpha_t}$ is the network’s implicit prediction of the clean image. With $\sigma_t = 0$ the update is deterministic and is exactly the probability-flow ODE step. With $\sigma_t = \sqrt{\tilde\beta_t}$ it recovers DDPM ancestral sampling. The interesting regime is $\sigma_t = 0$ with a sub-sampled timestep schedule: the same network, but you skip from $t = 1000$ to $t = 980$ to … to $t = 0$ in 50 large jumps. Sample quality is comparable to 1000-step DDPM and the cost is 20x smaller.

This is also why “consistency distillation” works: a deterministic sampler has a single trajectory from each noise sample, so you can train a student that maps $x_T$ directly to $x_0$ in one step.

Classifier-free guidance

Deriving the CFG update from the score view and the Bayesian gradient

Setup. Suppose we want to sample from $p(x \mid c)$ for some condition $c$ (a text prompt, a class label, an image map). By Bayes’ rule the conditional score is

\nabla_x \log p(x \mid c) = \nabla_x \log p(x) + \nabla_x \log p(c \mid x).

The first term is the unconditional score, available from $\varepsilon_\theta(x, t)$ . The second is a classifier gradient: it requires a separate model $p(c \mid x)$ trained at every noise level. Dhariwal & Nichol 2021 used this directly (“classifier guidance”) and got better samples by amplifying the classifier gradient by a factor $w$ :

\nabla_x \log \tilde p_w(x \mid c) = \nabla_x \log p(x) + (1 + w) \nabla_x \log p(c \mid x).

This works but requires training and storing a noise-conditional classifier on top of the diffusion model.

Ho & Salimans’s trick. Replace the explicit classifier with an implicit one. Train a single network that can act as both conditional and unconditional, by randomly dropping the conditioner $c$ during training (e.g., 10% of the time). At inference, you have access to both $\varepsilon_\theta(x_t, t, c)$ and $\varepsilon_\theta(x_t, t, \varnothing)$ , where $\varnothing$ is the null condition.

Using the score equivalence $\varepsilon_\theta \propto -\nabla_x \log p_t$ , the conditional and unconditional scores are

\nabla_x \log p(x \mid c) - \nabla_x \log p(x) = \nabla_x \log p(c \mid x).

The amplified-classifier score becomes

\nabla_x \log \tilde p_w(x \mid c) = \nabla_x \log p(x) + (1 + w) \bigl[\nabla_x \log p(x \mid c) - \nabla_x \log p(x)\bigr].

In $\varepsilon$ -space this is

\boxed{\ \tilde\varepsilon_\theta(x_t, t, c) = (1 + w)\, \varepsilon_\theta(x_t, t, c) - w\, \varepsilon_\theta(x_t, t, \varnothing).\ }

This is the classifier-free guidance update: compute conditional and unconditional noise predictions, extrapolate from unconditional to conditional with weight $1 + w$ , and use the result in any DDPM or DDIM step.

Behaviour. $w = 0$ recovers the conditional model; $w$ large pushes samples deeper into the high-conditional-density region, sharpening text adherence at the cost of diversity (mode dropping). Stable Diffusion uses $w \in [5, 9]$ at inference; values much higher than 15 produce oversaturated images, which is the signature of guidance pushing samples off the data manifold. CFG is the runtime knob that explains most of why one model can produce both “literal text adherence” and “diverse compositions” depending on how it is asked.

Train-test mismatch on conditioning drop rate

CFG requires the unconditional branch to be a real model, not a hack. If you set the drop rate too low (e.g. 1%), $\varepsilon_\theta(x, t, \varnothing)$ is undertrained and gives noisy unconditional predictions; the extrapolation $(1 + w) \varepsilon_c - w \varepsilon_\varnothing$ amplifies that noise. If you set it too high (e.g. 50%) the conditional branch is undertrained and CFG cannot recover. The DDPM-CFG default of 10% to 20% drop rate is a sensible starting point, but verify by plotting unconditional FID at $w = 0$ ; it should be reasonable, not catastrophically worse than conditional FID.

CFG is not free conditioning leakage protection

It is tempting to think that running with $w > 0$ “protects” the model from generating training-set leakage. It does the opposite: large $w$ collapses samples toward the modes of $p(x \mid c)$ , which for a finite training set means closer to memorised training points. If you care about not regurgitating training data, use a smaller $w$ , larger negative-prompt set, and dataset-deduplication, not bigger $w$ .

Latent diffusion

Pixel-space diffusion at $1024 \times 1024$ resolution is enormously expensive: each forward pass through the U-Net moves a $3 \times 1024 \times 1024 = 3 \times 10^6$ tensor through every layer, and you do it 1000 times per training step. Latent diffusion [7] attacks this directly:

Train an autoencoder (typically a VQ-GAN-style encoder-decoder) that maps pixels into a small latent space, e.g. $4 \times 64 \times 64 \approx 1.6 \times 10^4$ , and back. The encoder is fixed after this stage.
Run the entire diffusion model in latent space. The U-Net or DiT operates on $4 \times 64 \times 64$ tensors, not $3 \times 1024 \times 1024$ ones.
At inference, decode the final latent through the autoencoder to get a pixel image.

The latent space is roughly 64x smaller than the corresponding pixel space (factor of 8 in each spatial dimension), so training and inference are cheaper by approximately the same factor. Quality is comparable because the autoencoder removes high-frequency texture information that the diffusion model would otherwise spend most of its capacity modelling, while the diffusion model is responsible for the lower-frequency semantic content.

Stable Diffusion is the most prominent latent diffusion system; Stable Diffusion XL, Flux, and Stable Diffusion 3 are direct descendants with bigger nets and refined latent VAEs. The trade-off is that the autoencoder bottleneck imposes a quality ceiling: artifacts the encoder cannot represent (very fine text, complex reflections) cannot be generated by the model regardless of how well the diffusion is trained. Recent systems (SD3, Flux) use much larger latent VAEs with more channels to push this ceiling.

Architecture: U-Net and DiT

The reverse network $\varepsilon_\theta(x_t, t, c)$ takes a noised image (or latent), a timestep embedding, and an optional condition, and returns a tensor of the same shape as $x_t$ . Two architecture families dominate.

U-Net [8]. A symmetric encoder-decoder convolutional network with skip connections from each encoder block to the same-resolution decoder block. The skips let the network preserve spatial detail across the bottleneck, which is essential for image-to-image work where the reverse step only needs to make a small denoising correction. DDPM’s original network is a U-Net with attention layers added at the lower-resolution stages and the timestep injected via a learned sinusoidal embedding added to the residual blocks. This is the architecture behind Stable Diffusion 1/2.

DiT [9]. A plain Transformer over patch tokens of $x_t$ , with adaptive layer normalisation (adaLN) used to inject the timestep and class condition into every block. Patches are unembedded back to pixels at the end. DiT scales much better than U-Net: doubling parameters reliably improves FID, while U-Net plateaus. DiT is the architecture behind Stable Diffusion 3 and Sora; it is now the default for new large-scale systems.

The choice between U-Net and DiT is a scaling regime question. Below ~1B parameters, U-Net is competitive and cheaper to train. Above that, DiT wins.

Conditioning

Conditioning is “just another input to $\varepsilon_\theta$ ”. Three patterns dominate.

Class-conditional (small label set). Embed the class label as a vector and add it to the timestep embedding inside every residual block. Also used inside DiT’s adaLN.
Text-conditional (open-vocabulary). Encode the prompt with a frozen text encoder (CLIP or T5) into a sequence of token embeddings; have the U-Net’s attention layers cross-attend to this sequence, or have DiT prepend it as additional tokens. Cross-attention is the standard pattern in Stable Diffusion.
Image-conditional (img2img, ControlNet, inpainting). The conditioning image is concatenated channel-wise to $x_t$ at every block (img2img / inpainting), or fed through a parallel network whose features are added into the main U-Net (ControlNet). Both reduce to “extra inputs to $\varepsilon_\theta$ ”.

In all three cases CFG turns the conditioning strength into a runtime knob via the unconditional branch.

Comparison: diffusion vs GAN vs VAE vs autoregressive image

Property	Diffusion	GAN	VAE	Autoregressive image
Training stability	Plain MSE on a single network. Stable, monotonic loss curves.	Saddle-point minimax. Notoriously hard. Mode-collapse and discriminator overpowering common.	Reconstruction + KL. Stable but with a tendency to ignore the latent (posterior collapse).	Plain teacher-forced cross-entropy. Stable.
Mode coverage	High by construction; trained on full data density.	Often low. GANs easily collapse to subsets of modes.	High (KL term enforces it).	High; per-pixel likelihood is faithful.
Sample quality	SOTA for image, video, 3D as of 2024.	Was SOTA 2018-2020; now beaten on most benchmarks.	Typically blurry; the Gaussian decoder smooths fine detail.	Sharp but slow; quality scales with model size.
Sampling cost	High: 50-1000 forward passes per sample. Distillation reduces to 1-4.	Low: 1 forward pass.	Low: 1 forward pass.	Very high: O(N) sequential passes for N tokens.
Conditioning	Easy: any extra input. CFG is a runtime knob.	Possible (cGAN, StyleGAN). No clean text-conditioning until very late.	Possible (cVAE).	Native (just prepend to context).
Likelihood	Variational lower bound; tight enough for compression in some variants.	No likelihood (implicit model).	Variational lower bound; the standard tool for likelihood-based generative work.	Exact likelihood by chain rule. Clean.
See	GANs	—	VAEs	—

The summary is: diffusion replaced GANs as the SOTA image generator because training is stable, mode coverage is high, and conditioning is easy, even though sampling is more expensive. Most production image generators today are diffusion; the active frontier is on (a) sampling efficiency via distillation, (b) bigger latent spaces, and (c) replacing the U-Net with DiT-class Transformers.

Modern systems

Stable Diffusion 1/2/XL, Stable Diffusion 3, Flux, DALL-E 3, Imagen 3, and Midjourney are all latent diffusion models. Sora and Veo extend the same recipe to video by adding temporal patches. Audio diffusion (Riffusion, AudioLDM) runs on spectrogram latents. The recipe is the same in every case: a frozen perceptual encoder, a diffusion model in latent space, classifier-free guidance, a Transformer or DiT backbone, and text conditioning via cross-attention or prepended tokens.

Sampling cost and mitigations

The standard 50-step DDIM sampler is $\sim 50\times$ more expensive at inference than a single GAN forward pass. The active line of work to close that gap:

Higher-order solvers (DPM-Solver, DPM-Solver++, UniPC). Use the same trained $\varepsilon_\theta$ but with multi-step ODE solvers; reach DDIM-50 quality at 10-15 steps.
Consistency models (Song et al. 2023). Train a student that maps $x_t$ directly to $x_0$ for any $t$ ; sample in 1 step with quality close to teacher’s 50-step output.
Adversarial distillation (LCM-LoRA, SDXL-Turbo). Distil a teacher’s many-step trajectory into a 1-4-step student using a GAN-style adversarial loss for sharpness. Now the dominant approach for real-time image generation.

Build it yourself

A weekend project

A staged plan that takes diffusion from formula to working tiny model:

Implement DDPM forward sampling on MNIST. Plot $\bar\alpha_t / (1 - \bar\alpha_t)$ across $t$ to confirm the schedule.
Train a small U-Net (4 down / 4 up blocks, 64 channels) with $L_{\mathrm{simple}}$ . After ~30 minutes on a single GPU you should generate recognisable digits with the 1000-step ancestral sampler.
Implement DDIM with $\sigma_t = 0$ and a 50-step schedule; confirm samples are comparable.
Add class conditioning (label embedding added to the timestep embedding) and CFG; sweep $w \in \{0, 1, 3, 7\}$ and look at the digit-fidelity / diversity trade-off.
Stretch: replace pixel diffusion with latent diffusion. Train a tiny VAE on MNIST first, then run the same DDPM in 4-channel $7 \times 7$ latent space. The sample-cost improvement is small at MNIST scale but the architecture change is the point.
Stretch: implement a 4-step consistency distillation of your teacher and compare 1-step and 4-step student samples to teacher.

Reference implementations: denoising-diffusion-pytorch and Hugging Face Diffusers.

Quick code

# Tiny PyTorch DDPM training step. Assumes a U-Net `model(x_t, t, c=None)`.
import torch
import torch.nn.functional as F
 
T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1.0 - betas
alpha_bar = torch.cumprod(alphas, dim=0)
 
def diffusion_step(model, x0, c=None, p_uncond=0.1):
    B = x0.size(0)
    t = torch.randint(0, T, (B,), device=x0.device)
    a_bar = alpha_bar.to(x0.device)[t].view(B, 1, 1, 1)
    eps = torch.randn_like(x0)
    x_t = a_bar.sqrt() * x0 + (1 - a_bar).sqrt() * eps
    # Classifier-free guidance: drop the conditioner with probability p_uncond.
    if c is not None and torch.rand(()) < p_uncond:
        c_in = None
    else:
        c_in = c
    eps_pred = model(x_t, t, c=c_in)
    return F.mse_loss(eps_pred, eps)

# DDIM sampler, 50 steps, deterministic (sigma_t = 0).
@torch.no_grad()
def ddim_sample(model, shape, c=None, w=7.0, n_steps=50, device='cuda'):
    x = torch.randn(shape, device=device)
    ts = torch.linspace(T - 1, 0, n_steps + 1).long().to(device)
    for i in range(n_steps):
        t, t_next = ts[i], ts[i + 1]
        a_bar = alpha_bar.to(device)[t]
        a_bar_next = alpha_bar.to(device)[t_next] if t_next >= 0 else torch.tensor(1.0, device=device)
        if c is None or w == 0:
            eps = model(x, t.expand(x.size(0)), c=c)
        else:
            # Classifier-free guidance: extrapolate (1 + w) * eps_c - w * eps_uncond.
            eps_c = model(x, t.expand(x.size(0)), c=c)
            eps_u = model(x, t.expand(x.size(0)), c=None)
            eps = (1 + w) * eps_c - w * eps_u
        x0_hat = (x - (1 - a_bar).sqrt() * eps) / a_bar.sqrt()
        x = a_bar_next.sqrt() * x0_hat + (1 - a_bar_next).sqrt() * eps
    return x

Pop quiz

Check yourself

Score: 0 / 5(0 answered)

1. In DDPM, why does L_simple drop the per-timestep weighting beta_t^2 / (2 sigma_t^2 alpha_t (1 - alpha_bar_t))?
2. The closed-form q(x_t | x_0) in DDPM is N(sqrt(alpha_bar_t) x_0, (1 - alpha_bar_t) I). What is the consequence for training?
3. What does classifier-free guidance compute at sampling time, given conditional and unconditional eps networks?
4. Why is latent diffusion (Stable Diffusion) much cheaper than pixel-space diffusion at the same image resolution?
5. Which property best explains why diffusion replaced GANs as the SOTA image generator?

Interview questions

Derive the DDPM variational lower bound and show how it reduces to the simple noise-prediction MSE.research

Treat $x_{1:T}$ as latents. Apply Jensen’s inequality to the negative log-likelihood:

-\log p_\theta(x_0) \le \mathbb{E}_q\!\left[ -\log \frac{p_\theta(x_{0:T})}{q(x_{1:T} \mid x_0)} \right] =: L_{\mathrm{vlb}}.

Factor $q$ and $p_\theta$ as Markov chains and rewrite each forward $q(x_t \mid x_{t-1})$ via Bayes’ rule conditioned on $x_0$ . After telescoping, the bound regroups into

L_{\mathrm{vlb}} = L_T + \sum_{t=2}^{T} L_{t-1} + L_0,

with $L_T = D_{\mathrm{KL}}(q(x_T \mid x_0) \,\|\, p(x_T))$ a constant, $L_0$ a discrete decoder term handled separately, and $L_{t-1} = D_{\mathrm{KL}}(q(x_{t-1} \mid x_t, x_0) \,\|\, p_\theta(x_{t-1} \mid x_t))$ .

Both Gaussians in $L_{t-1}$ have the same fixed variance $\sigma_t^2$ , so the KL collapses to $\lVert \tilde\mu_t(x_t, x_0) - \mu_\theta(x_t, t) \rVert^2 / (2 \sigma_t^2)$ plus a constant. Substitute $x_0 = (x_t - \sqrt{1 - \bar\alpha_t}\, \varepsilon) / \sqrt{\bar\alpha_t}$ from the closed-form noising; the forward posterior mean simplifies to $\tilde\mu_t = (1/\sqrt{\alpha_t}) (x_t - \beta_t / \sqrt{1 - \bar\alpha_t}\, \varepsilon)$ . Parameterise the reverse mean in the same form with $\varepsilon_\theta(x_t, t)$ replacing $\varepsilon$ . The squared distance reduces to a constant times $\lVert \varepsilon - \varepsilon_\theta(x_t, t) \rVert^2$ . Dropping the per- $t$ constant gives

L_{\mathrm{simple}} = \mathbb{E}_{t, x_0, \varepsilon} \,\lVert \varepsilon - \varepsilon_\theta(\sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \varepsilon, t) \rVert^2.

That is the entire training loss.

Why is epsilon-prediction equivalent to score matching? State the precise scaling.research

By Tweedie’s formula, for $y = x + \sigma z$ with $z \sim \mathcal{N}(0, I)$ , the posterior mean is $\mathbb{E}[x \mid y] = y + \sigma^2 \nabla_y \log p_\sigma(y)$ . Apply it to the DDPM marginal $q(x_t \mid x_0) = \mathcal{N}(\sqrt{\bar\alpha_t}\, x_0, (1 - \bar\alpha_t) I)$ :

\mathbb{E}[\sqrt{\bar\alpha_t}\, x_0 \mid x_t] = x_t + (1 - \bar\alpha_t) \nabla_{x_t} \log q(x_t).

But $\sqrt{\bar\alpha_t}\, x_0 = x_t - \sqrt{1 - \bar\alpha_t}\, \varepsilon$ , so $\mathbb{E}[\varepsilon \mid x_t] = -\sqrt{1 - \bar\alpha_t} \nabla_{x_t} \log q(x_t)$ .

The Bayes-optimal $\varepsilon_\theta(x_t, t)$ is therefore $-\sqrt{1 - \bar\alpha_t}$ times the true score. Training $\varepsilon_\theta$ with MSE is denoising score matching; the two parameterisations differ only by the known scaling $\sqrt{1 - \bar\alpha_t}$ . This is why a model trained with DDPM’s loss can be sampled with any score-based method: ancestral, DDIM, probability-flow ODE, Langevin, higher-order solvers, or 1-step consistency students. Same network, many discretisations.

Derive classifier-free guidance from the Bayesian decomposition of the conditional score.research

The conditional score factors as

\nabla_x \log p(x \mid c) = \nabla_x \log p(x) + \nabla_x \log p(c \mid x).

Classifier guidance amplifies the second term by a factor $1 + w$ , sampling instead from a tilted distribution whose score is $\nabla_x \log p(x) + (1 + w) \nabla_x \log p(c \mid x)$ . Using the score equivalence and the additive decomposition, this rewrites as

\nabla_x \log \tilde p_w(x \mid c) = \nabla_x \log p(x) + (1 + w) \bigl[\nabla_x \log p(x \mid c) - \nabla_x \log p(x)\bigr].

In $\varepsilon$ -space (using the negative-scaled equivalence above):

\tilde\varepsilon = (1 + w)\, \varepsilon_\theta(x_t, t, c) - w\, \varepsilon_\theta(x_t, t, \varnothing).

This is a runtime extrapolation between the conditional and unconditional predictions of the same network, which is what makes it “classifier-free”: you do not train a separate $p(c \mid x)$ classifier. To make it work, train $\varepsilon_\theta$ with the conditioner randomly dropped (10-20%) so the unconditional branch is a real model. CFG’s $w$ knob explains most of why diffusion text-to-image systems can be tuned at inference for either literal text adherence (high $w$ ) or compositional diversity (low $w$ ).

Why is latent diffusion much cheaper than pixel diffusion at the same output resolution? Give a concrete cost ratio.onsite

A diffusion model’s per-step compute scales with the size of the tensor $\varepsilon_\theta$ operates on (and the architecture’s per-element FLOPs, roughly fixed across U-Nets of a given width). Pixel-space SD-1 at $512 \times 512$ would denoise a $3 \cdot 512 \cdot 512 \approx 7.9 \times 10^5$ -element tensor; latent-space SD-1 denoises a $4 \cdot 64 \cdot 64 \approx 1.6 \times 10^4$ -element tensor. The ratio is approximately $48\times$ . Multiply by 1000 training steps per epoch, and the training cost difference is the same factor.

The catch is that you have to amortise the cost of training the autoencoder, and you inherit a quality ceiling from its reconstruction error: artifacts the encoder cannot represent (very fine text, small repeated patterns) cannot be recovered by the diffusion model, regardless of how well it models the latent distribution. Stable Diffusion 3 and Flux push this ceiling by using larger autoencoders with more latent channels (16 instead of 4), accepting somewhat higher compute in exchange for better fine-grained quality.

The economic consequence is that essentially every production text-to-image model since 2022 is a latent diffusion model. Pixel-space diffusion survives in low-resolution research settings (CIFAR, Tiny-ImageNet) and in cascaded pipelines (Imagen) where a low-resolution pixel-space model is followed by a super-resolution diffusion model.

Interactive Demo

Diffusion Process Visualizer

Watch how a smiley face is gradually destroyed by noise (forward) or recovered from noise (reverse).

Noise step: 0 / 20

Step

Noise level β

0.00

State

Slightly noisy - structure still clear

What to try: Drag the slider to add or remove noise one step at a time

References

Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. ICML 2015. arXiv:1503.03585
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239
Song, Y. and Ermon, S. (2019). Generative Modeling by Estimating Gradients of the Data Distribution. NeurIPS 2019. arXiv:1907.05600
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2020). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021 (arXiv preprint 2020). arXiv:2011.13456
Song, J., Meng, C., and Ermon, S. (2020). Denoising Diffusion Implicit Models. ICLR 2021 (arXiv preprint 2020). arXiv:2010.02502
Ho, J. and Salimans, T. (2022). Classifier-Free Diffusion Guidance. NeurIPS 2021 Workshop (arXiv preprint 2022). arXiv:2207.12598
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2021). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022 (arXiv preprint 2021). arXiv:2112.10752
Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI 2015. arXiv:1505.04597
Peebles, W. and Xie, S. (2022). Scalable Diffusion Models with Transformers. ICCV 2023 (arXiv preprint 2022). arXiv:2212.09748
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org

Agents & Tool Use Multimodal Models