Generative Adversarial Networks

Background

Goodfellow et al. (2014) introduced GANs as a minimax two-player game between generator and discriminator
Produced the first photorealistic synthetic face images (ProGAN 2018, StyleGAN 2019)
Training instability (mode collapse, vanishing gradients) drove years of research in GAN stabilization

The Concept

A GAN trains two networks in opposition: the generator creates fake samples trying to fool a discriminator; the discriminator tries to distinguish real from fake. The generator improves as the discriminator improves, reaching a Nash equilibrium where fakes are indistinguishable from real samples.

How it Links

Diffusion models have largely replaced GANs for image generation due to more stable training
The adversarial training idea lives on in adversarial robustness, domain adaptation, and RLHF (discriminator = reward model)
Progressive growing (ProGAN) connects to curriculum learning in neural network training

Concrete Example

Generating faces: the generator starts producing blurry noise. The discriminator easily distinguishes these from real faces. The generator improves, producing blurry faces. The discriminator adapts. After thousands of rounds, the generator produces photorealistic faces that fool both humans and the discriminator - but they are entirely synthetic.

Prerequisites

Multi-Layer Perceptron, Convolutional Neural Networks, Autoencoders & VAEs, Math you need: probability & calculus

Learning outcomes

State the original GAN minimax objective and explain why it is a two-player zero-sum game with no monotone Lyapunov function.
Derive the optimal discriminator D*(x) = p_data(x) / (p_data(x) + p_g(x)) from the value function and substitute back to recover 2 * JSD(p_data || p_g) - log 4.
Explain why Jensen-Shannon divergence has zero gradient when the supports of p_data and p_g do not overlap, motivating the move to Wasserstein distance.
State the Kantorovich-Rubinstein duality for the Earth Mover distance and explain how it turns W into a sup over 1-Lipschitz critics.
Compare weight clipping (WGAN) with the gradient penalty (WGAN-GP), and write the gradient-penalty regularizer.
Sketch the architectural progression DCGAN -> Progressive GAN -> StyleGAN/StyleGAN2 -> BigGAN, and name the key trick at each step.
Explain mode collapse, how to detect it, and why FID is the standard evaluation metric (Heusel 2017).
Explain why diffusion displaced GANs as the state of the art for image generation around 2020-2022.

TL;DR

A GAN trains two networks against each other: a generator $G(z)$ that maps noise $z \sim p_z$ to a sample, and a discriminator $D(x)$ that classifies real vs. fake. The original objective is the minimax game [1]

\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))].

With $G$ fixed, the optimal discriminator is $D^*(x) = p_{\text{data}}(x) / (p_{\text{data}}(x) + p_g(x))$ , and substituting it back gives $2 \mathrm{JSD}(p_{\text{data}} \,\Vert\, p_g) - \log 4$ , so the global optimum is at $p_g = p_{\text{data}}$ . Training is unstable because Jensen-Shannon goes flat (zero gradient) when the supports of $p_{\text{data}}$ and $p_g$ do not overlap. WGAN [4] swaps JSD for the Earth Mover (Wasserstein) distance, which has well-defined gradients almost everywhere; WGAN-GP [5] replaces weight clipping with a gradient penalty. DCGAN [3], Progressive GAN [6], and StyleGAN/StyleGAN2 [7][8] drove image quality from 64x64 toys to photorealistic 1024x1024 faces. Conditional GAN [2] conditions both networks on a label, and CycleGAN [10] handles unpaired domain translation. GANs were SOTA for image generation 2015-2020 and were largely displaced by diffusion [12] after 2020 because diffusion has a stable, single-network training objective.

Intuition

Think of a GAN as a counterfeiter and a detective. The counterfeiter (generator) prints fake money; the detective (discriminator) sees a stack of bills, half real and half fake, and tries to label each. Each round, the detective gets a little better at spotting fakes, and the counterfeiter gets a little better at making fakes that the detective cannot distinguish from real. At equilibrium, the detective is reduced to a coin flip and the counterfeiter is producing bills indistinguishable from the central bank’s.

The math says exactly this. Define the value function

V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))].

The discriminator wants to maximize $V$ (high $D(x)$ on real samples, low $D(G(z))$ on fakes). The generator wants to minimize $V$ (push $D(G(z))$ up). The training procedure alternates a few discriminator gradient ascent steps with one generator gradient descent step. Unlike an autoencoder there is no reconstruction target; the only signal the generator ever sees is the discriminator’s gradient.

Why this is fundamentally harder than supervised learning

In supervised learning the loss is a fixed function of the parameters, so the gradient field is a vector field on parameter space, and gradient descent monotonically decreases the loss. In a GAN the discriminator’s loss surface depends on the generator and vice versa: both networks are climbing a moving target. There is no scalar Lyapunov function whose value monotonically decreases over training, so loss curves are not interpretable in the usual way. The training is a saddle-point optimization, and convergence is only guaranteed in restrictive theoretical settings.

The practical consequence is that GAN training requires careful tricks (learning-rate schedules, alternating step counts, batch normalization choices, gradient clipping, spectral normalization). Loss curves look noisy and oscillating even on a converging run; the only reliable health signal is sample quality and FID [11].

The optimal discriminator and the JSD interpretation

Deriving D*(x) and the JSD form (interview gold)

Setup

Fix the generator $G$ , so $p_g$ is the implied distribution of $G(z)$ for $z \sim p_z$ . Treat the value function as a functional of $D$ alone:

V(D, G) = \int_x p_{\text{data}}(x) \log D(x)\, dx + \int_z p_z(z) \log(1 - D(G(z)))\, dz.

Apply the change-of-variables $x = G(z)$ to the second integral so both are integrals over $x$ :

V(D, G) = \int_x \bigl( p_{\text{data}}(x) \log D(x) + p_g(x) \log(1 - D(x)) \bigr)\, dx.

Step 1: maximize $D$ pointwise

For each fixed $x$ , the integrand is $a \log y + b \log(1 - y)$ with $a = p_{\text{data}}(x) \ge 0$ , $b = p_g(x) \ge 0$ , and $y = D(x) \in [0, 1]$ . Differentiate with respect to $y$ and set to zero:

\frac{a}{y} - \frac{b}{1 - y} = 0 \;\Longrightarrow\; y^* = \frac{a}{a + b}.

Therefore the optimal discriminator at any point $x$ where $p_{\text{data}}(x) + p_g(x) > 0$ is

\boxed{\; D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}. \;}

This is intuitive: the Bayes-optimal classifier for “real vs. fake” mixed in a 50/50 prior is the posterior probability of the “real” class, which is exactly the ratio above.

Step 2: substitute $D^*$ back into $V$

Plug $D^*$ into $V(D, G)$ :

V(D^*, G) = \int_x p_{\text{data}}(x) \log \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)} + p_g(x) \log \frac{p_g(x)}{p_{\text{data}}(x) + p_g(x)}\, dx.

Add and subtract $\log 2$ in each term to massage the integrand:

\log \frac{p_{\text{data}}}{p_{\text{data}} + p_g} = \log \frac{p_{\text{data}}}{(p_{\text{data}} + p_g) / 2} - \log 2.

Let $m = \tfrac{1}{2}(p_{\text{data}} + p_g)$ be the mid-distribution. Then

V(D^*, G) = -\log 4 + \mathrm{KL}(p_{\text{data}} \,\Vert\, m) + \mathrm{KL}(p_g \,\Vert\, m).

The Jensen-Shannon divergence is by definition

\mathrm{JSD}(p \,\Vert\, q) = \tfrac{1}{2} \mathrm{KL}(p \,\Vert\, m) + \tfrac{1}{2} \mathrm{KL}(q \,\Vert\, m), \quad m = \tfrac{1}{2}(p + q),

\boxed{\; V(D^*, G) = 2\,\mathrm{JSD}(p_{\text{data}} \,\Vert\, p_g) - \log 4. \;}

Step 3: read off the global optimum

JSD is non-negative, with $\mathrm{JSD}(p \,\Vert\, q) = 0$ iff $p = q$ almost everywhere. Therefore the generator’s objective $\min_G V(D^*, G)$ is minimized at $p_g = p_{\text{data}}$ , where $V(D^*, G) = -\log 4$ . In words: at the global optimum, $G$ has matched the data distribution exactly, and $D^*$ outputs $\tfrac{1}{2}$ everywhere, the random-guess baseline.

Step 4: why this fails in practice

The trouble is that $\mathrm{JSD}(p_{\text{data}} \,\Vert\, p_g)$ is constant (equal to $\log 2$ ) whenever the supports of $p_{\text{data}}$ and $p_g$ do not overlap on a set of positive measure. For early training, $p_g$ is concentrated on a low-dimensional manifold disjoint from the data manifold, so JSD is at the ceiling and its gradient with respect to $G$ is zero. The discriminator is too good, and the generator gets no learning signal: the loss is high, the gradient is zero, training stalls. This is the vanishing-gradient failure mode that motivated Wasserstein GAN [4].

The non-saturating generator loss

Goodfellow’s original paper proposed a practical fix even before WGAN: instead of the generator minimizing $\log(1 - D(G(z)))$ , have it maximize $\log D(G(z))$ . The two are equivalent at the saddle point but the non-saturating form has much larger gradients when $D(G(z))$ is small (early training), so the generator is not stuck. Most production code uses this non-saturating loss, even though textbooks present the original minimax form for the JSD derivation [1][13].

Mode collapse

A GAN can find a degenerate equilibrium where the generator outputs a small set of points (in the worst case, one point) that fools the current discriminator. The discriminator then adapts, the generator jumps to a different small set, and so on. The samples are sharp and pass the discriminator but the diversity of the data is lost.

Mode collapse is hard to detect from training loss alone (loss can look fine). Standard tooling:

Inception Score (IS): use a fixed pretrained classifier and compute $\exp(\mathbb{E}_x[\mathrm{KL}(p(y \mid x) \,\Vert\, p(y))])$ . Higher means each sample is confidently classified and the marginal label distribution is uniform. Mode-collapsed generators have low IS because the marginal collapses.
FID (Frechet Inception Distance) [11]: extract features from a pretrained Inception-v3, fit a Gaussian to real and fake feature distributions, and compute $\lVert \mu_r - \mu_g \rVert_2^2 + \mathrm{tr}(\Sigma_r + \Sigma_g - 2 (\Sigma_r \Sigma_g)^{1/2})$ . Lower is better. FID is the standard metric for GAN evaluation since 2017 and tracks both quality and diversity.
Visual diversity audit: sample a large batch, cluster, and check that the cluster count and per-cluster spread match the data. A 10-mode mixture should give roughly 10 generator modes.

Wasserstein GAN: from JSD to Earth Mover

Wasserstein distance and Kantorovich-Rubinstein duality

The Earth Mover distance

The Wasserstein-1 (Earth Mover) distance between two distributions on $\mathbb{R}^d$ is

W(p, q) = \inf_{\gamma \in \Pi(p, q)} \mathbb{E}_{(x, y) \sim \gamma}[\, \lVert x - y \rVert\,],

where $\Pi(p, q)$ is the set of all joint distributions $\gamma$ with marginals $p$ and $q$ . Intuitively, $\gamma(x, y)$ is a transport plan that moves probability mass from $x$ to $y$ , and $W$ is the minimum total work (mass times distance) needed to morph $p$ into $q$ .

Unlike JSD, $W(p, q)$ is well-defined and continuous even when the supports do not overlap. If $p$ is a delta at $0$ and $q$ is a delta at $\theta$ , then $\mathrm{JSD}(p \,\Vert\, q) = \log 2$ for any $\theta \neq 0$ (a constant cliff), but $W(p, q) = |\theta|$ (a smooth slope). The smooth slope is exactly the gradient signal that GAN training needs.

Kantorovich-Rubinstein duality

The infimum over transport plans is hard to compute directly. The KR duality theorem rewrites it as a supremum over 1-Lipschitz functions $f$ :

\boxed{\; W(p, q) = \sup_{\lVert f \rVert_L \le 1} \mathbb{E}_{x \sim p}[f(x)] - \mathbb{E}_{y \sim q}[f(y)]. \;}

Here $\lVert f \rVert_L$ is the Lipschitz constant of $f$ , i.e. $|f(x) - f(y)| \le \lVert x - y \rVert$ for all $x, y$ . The 1-Lipschitz constraint is what couples the two expectations: an unconstrained $f$ could just diverge.

From duality to the WGAN objective

Replace the JSD-style discriminator with a critic $f_w$ (a neural net with parameters $w$ ) that we constrain to be 1-Lipschitz. The training objective becomes

\min_G \max_{w :\, \lVert f_w \rVert_L \le 1} \mathbb{E}_{x \sim p_{\text{data}}}[f_w(x)] - \mathbb{E}_{z \sim p_z}[f_w(G(z))].

The critic does not output a probability; it outputs a real-valued score. The generator gets a clean gradient as long as $f_w$ is approximately 1-Lipschitz, even when the supports of $p_g$ and $p_{\text{data}}$ are disjoint.

Enforcing the Lipschitz constraint: clipping vs. gradient penalty

Original WGAN [4] enforces the constraint by weight clipping: after each critic update, project all weights into a box $[-c, c]$ with $c$ a small constant (e.g. 0.01). This is crude. The clipping value has to be tuned, the clipped weights tend to saturate at the boundary (so the critic lives in a narrow function class), and the resulting Lipschitz constant is implicit and not exactly 1.

WGAN-GP [5] replaces clipping with a gradient penalty: add a soft penalty that pushes the critic’s gradient norm toward 1 on points sampled along straight lines between real and fake samples. The full objective is

\mathcal{L}_{\text{critic}} = \mathbb{E}_{z}[f_w(G(z))] - \mathbb{E}_{x}[f_w(x)] + \lambda \, \mathbb{E}_{\hat x}\!\left[ ( \lVert \nabla_{\hat x} f_w(\hat x) \rVert_2 - 1)^2 \right],

with $\hat x = \alpha x + (1 - \alpha) G(z)$ for $\alpha \sim U[0, 1]$ and typical $\lambda = 10$ . The penalty is a two-sided penalty toward unit gradient norm: an exact 1-Lipschitz function has unit gradient norm a.e., and the squared term penalizes deviations on either side of $1$ , making that the network’s preferred operating point. Empirically WGAN-GP trains more stably than WGAN clipping across many architectures, and it scales to larger models without retuning $\lambda$ .

What the critic actually estimates

By KR duality the critic’s value $\max_w (\mathbb{E}_p[f_w] - \mathbb{E}_q[f_w])$ approaches $W(p, q)$ from below as the critic’s capacity grows, so the critic loss curve does correlate with $W$ . This is unlike the original GAN where the discriminator’s loss is uninformative as a quality metric. WGAN gives a useful health signal during training.

Architectural progression

A short history of the empirical breakthroughs.

2014Original GAN (Goodfellow et al.)breakthrough
Two MLPs, minimax JSD objective, MNIST and CIFAR. Showed the idea works at all; samples were small and noisy.
2014Conditional GAN (Mirza & Osindero)method
Concatenate a label y to both G and D inputs. The same objective, but now G samples class-conditional images.
2015DCGAN (Radford, Metz, Chintala)architecture
Replace MLPs with strided / fractionally-strided convs, add BatchNorm, drop pooling. First recipe for stable image GAN training. 64x64 bedrooms, faces.
2017WGAN / WGAN-GP (Arjovsky 2017, Gulrajani 2017)method
Earth Mover loss with weight clipping or a gradient penalty. Stable training across many architectures, useful loss curves.
2017Progressive GAN (Karras et al.)architecture
Grow generator and discriminator from 4x4 to 1024x1024 by adding layers during training. First photorealistic GAN faces (CelebA-HQ).
2017CycleGAN (Zhu et al.)application
Two GANs and a cycle-consistency loss let G learn an unpaired domain translation: horses to zebras, summer to winter, photo to Monet.
2018BigGAN (Brock, Donahue, Simonyan)architecture
Scale up class-conditional GANs on ImageNet with truncation, orthogonal regularization, and large batches. State-of-the-art FID at scale.
2019StyleGAN / StyleGAN2 (Karras et al.)architecture
Style-based generator: a learned mapping network produces per-layer style codes via AdaIN. StyleGAN2 fixes blob artifacts with weight demodulation. Photorealistic 1024x1024 faces, the high-water mark of GAN image quality.
2020DDPM (Ho, Jain, Abbeel)displaced
Denoising Diffusion Probabilistic Models: a single network, stable likelihood-based training, mode-covering. Within a year, diffusion overtakes GANs on ImageNet FID and never gives the lead back.

StyleGAN in one paragraph

StyleGAN [7] redesigns the generator: instead of feeding $z$ into the first layer, a fully-connected mapping network transforms $z$ into an intermediate latent $w$ , and at each resolution a learned affine projects $w$ to per-channel $\gamma, \beta$ that modulate the activations via Adaptive Instance Normalization. The first layer takes a learned constant tensor, not noise. Stochastic per-layer noise is added separately for fine details. The result is a hierarchy where coarse layers control pose, middle layers control hair/face shape, and fine layers control texture and color. StyleGAN2 [8] replaces AdaIN with weight demodulation (rescaling conv weights instead of activations), eliminating the characteristic “droplet” artifacts. StyleGAN2 on FFHQ-1024 was the visual benchmark the field chased for years.

Conditional GAN and CycleGAN

Conditional GAN [2] simply concatenates a one-hot label $y$ (or any conditioning signal) to both $G$ ‘s input and $D$ ‘s input. The objective is unchanged; only the inputs grow. This is how class-conditional sampling and text-conditional GANs work.

CycleGAN [10] tackles unpaired domain translation: you have a stack of horse photos and a stack of zebra photos, but no aligned pairs. Train two generators $G : H \to Z$ and $F : Z \to H$ along with two discriminators, plus a cycle-consistency loss $\lVert F(G(h)) - h \rVert_1 + \lVert G(F(z)) - z \rVert_1$ that says you should be able to round-trip through both translators back to the original. This forces $G$ and $F$ to be approximate inverses, which in turn prevents either from collapsing onto a degenerate mapping.

Comparison

Variant	Objective	D / critic constraint	Training stability	Image quality (FID, lower is better)	Year
Original GAN	minimax JSD (2 * JSD - log 4)	D outputs prob. in [0, 1] via sigmoid	unstable; mode collapse common; loss uninformative	~50+ on small datasets	2014
DCGAN	same as GAN, non-saturating G loss	sigmoid + BatchNorm, strided convs	much better than vanilla; still tricky on hard data	~30 on LSUN bedrooms 64x64	2015
WGAN (clip)	Earth Mover via KR duality	1-Lipschitz via weight clipping	stable; loss correlates with W; slow critic convergence	similar to DCGAN, more reliable	2017
WGAN-GP	Earth Mover with gradient penalty	1-Lipschitz via grad-norm penalty (lambda ~ 10)	most stable across architectures	beats WGAN clip; standard GAN baseline	2017
Progressive GAN	WGAN-GP loss	gradient penalty	stable up to 1024x1024 via progressive growing	photorealistic CelebA-HQ; FID under 10	2017
StyleGAN2	non-saturating GAN with R1 regularizer	R1 grad penalty on real samples	stable, the empirical state of the art for GANs	FID ~3 on FFHQ-1024; visual peak of GANs	2020

Implementations

A canonical DCGAN sketch in PyTorch and the Keras equivalent. The architecture is small enough to read in one screen but large enough to actually train on 64x64 images.

python

import torch
import torch.nn as nn
import torch.nn.functional as F

NZ = 100   # latent dim
NGF = 64   # generator feature multiplier
NDF = 64   # discriminator feature multiplier
NC = 3     # output channels (RGB)

class Generator(nn.Module):
  """DCGAN generator: noise -> 64x64 RGB image via fractionally-strided convs."""

  def __init__(self) -> None:
      super().__init__()
      self.net = nn.Sequential(
          # 1x1 -> 4x4
          nn.ConvTranspose2d(NZ, NGF * 8, 4, 1, 0, bias=False),
          nn.BatchNorm2d(NGF * 8),
          nn.ReLU(True),
          # 4x4 -> 8x8
          nn.ConvTranspose2d(NGF * 8, NGF * 4, 4, 2, 1, bias=False),
          nn.BatchNorm2d(NGF * 4),
          nn.ReLU(True),
          # 8x8 -> 16x16
          nn.ConvTranspose2d(NGF * 4, NGF * 2, 4, 2, 1, bias=False),
          nn.BatchNorm2d(NGF * 2),
          nn.ReLU(True),
          # 16x16 -> 32x32
          nn.ConvTranspose2d(NGF * 2, NGF, 4, 2, 1, bias=False),
          nn.BatchNorm2d(NGF),
          nn.ReLU(True),
          # 32x32 -> 64x64
          nn.ConvTranspose2d(NGF, NC, 4, 2, 1, bias=False),
          nn.Tanh()    # outputs in [-1, 1]
      )

  def forward(self, z):
      return self.net(z.view(z.size(0), NZ, 1, 1))


class Discriminator(nn.Module):
  """DCGAN discriminator: 64x64 RGB -> scalar prob via strided convs."""

  def __init__(self) -> None:
      super().__init__()
      self.net = nn.Sequential(
          nn.Conv2d(NC, NDF, 4, 2, 1, bias=False),
          nn.LeakyReLU(0.2, inplace=True),
          nn.Conv2d(NDF, NDF * 2, 4, 2, 1, bias=False),
          nn.BatchNorm2d(NDF * 2),
          nn.LeakyReLU(0.2, inplace=True),
          nn.Conv2d(NDF * 2, NDF * 4, 4, 2, 1, bias=False),
          nn.BatchNorm2d(NDF * 4),
          nn.LeakyReLU(0.2, inplace=True),
          nn.Conv2d(NDF * 4, NDF * 8, 4, 2, 1, bias=False),
          nn.BatchNorm2d(NDF * 8),
          nn.LeakyReLU(0.2, inplace=True),
          nn.Conv2d(NDF * 8, 1, 4, 1, 0, bias=False),
          nn.Sigmoid()
      )

  def forward(self, x):
      return self.net(x).view(-1)


# Training loop sketch (non-saturating G loss).
# G, D = Generator().to(dev), Discriminator().to(dev)
# optG = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
# optD = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
# bce = nn.BCELoss()
# for x, _ in loader:
#     x = x.to(dev)
#     bs = x.size(0)
#     # ---- D step ----
#     z = torch.randn(bs, NZ, device=dev)
#     fake = G(z).detach()
#     loss_d = bce(D(x), torch.ones(bs, device=dev)) + bce(D(fake), torch.zeros(bs, device=dev))
#     optD.zero_grad(); loss_d.backward(); optD.step()
#     # ---- G step (non-saturating: maximize log D(G(z)) ) ----
#     z = torch.randn(bs, NZ, device=dev)
#     loss_g = bce(D(G(z)), torch.ones(bs, device=dev))
#     optG.zero_grad(); loss_g.backward(); optG.step()

DCGAN recipe: ConvTranspose2d in G, strided Conv2d in D, BatchNorm, LeakyReLU(0.2), Adam(lr=2e-4, betas=(0.5, 0.999)), Tanh on G output, Sigmoid on D output.

Illustrative training curves

GAN training is too heavy for the in-browser Pyodide runner used elsewhere in this site (a real DCGAN run is on the order of GPU-hours, far beyond a WebAssembly Python interpreter). The plot below shows illustrative generator and critic loss curves from a typical WGAN-GP training run on CIFAR-10. The numbers are hard-coded for visualization; treat them as a sketch of what the loss curves look like, not as data from an actual run.

The numbers below are illustrative, not from a measured run

A real GAN training log shows the discriminator loss bouncing around a small range while the generator loss slowly trends down, with occasional excursions when the two networks “trade” advantage. The critic loss in WGAN-GP correlates monotonically with the Wasserstein distance, which is one of the reasons WGAN-GP is easier to debug than vanilla GAN. The numbers in the plot are picked to illustrate this qualitative shape.

Real-world uses

Domain	Use	Notes
Image generation	unconditional sample synthesis (faces, scenes, art)	StyleGAN2 / StyleGAN3 was the visual peak; on FFHQ-1024 GANs still produce sharper textures than equally-sized diffusion models, though diffusion has caught up.
Super-resolution	SRGAN, ESRGAN: 4x upscaling with perceptual + adversarial loss	Adversarial loss pushes outputs toward the manifold of real high-resolution images, fixing the over-smooth artifact of pure L2.
Image-to-image translation	pix2pix (paired), CycleGAN (unpaired): style transfer, season swap, photo-to-sketch	Conditional GAN with U-Net generator and PatchGAN discriminator; CycleGAN extends to unpaired data via cycle consistency.
Data augmentation	synthesize labeled training data for downstream classifiers	Works well in narrow domains (medical X-rays, faces) where collecting real data is expensive; gains on broad benchmarks are mixed.
Inpainting / editing	fill in masked regions of an image; latent-space editing of StyleGAN faces	StyleGAN`s W and W+ spaces support semantic edits (age, smile, gender) by linear directions found via supervised attribute classifiers.

Why diffusion replaced GANs

By 2022 most state-of-the-art image generation systems were diffusion-based, not GAN-based. The reasons are structural, not just incremental:

Single-network training. Diffusion trains one network with a stable per-step regression objective ( $\mathbb{E}[\lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2]$ ), with no adversary to balance. Loss curves are interpretable and debuggable like ordinary supervised learning.
Mode coverage by construction. A diffusion model is trained to denoise samples from the full data distribution, so it cannot collapse to a small subset of modes. GANs need explicit regularization (mini-batch discrimination, spectral norm) to avoid collapse, and even then it is a recurring problem.
Likelihood-based training. Diffusion has a tractable variational lower bound on log-likelihood and a clean theoretical grounding in stochastic differential equations. GANs do not have a likelihood; FID is the only widely-trusted metric.
Easier conditioning. Classifier-free guidance gives clean knobs for text-to-image and class-conditional generation; cGANs need re-architecting to support new conditioning modalities.
Trade-off: diffusion is slow at sample time (10-50+ network passes per sample). GANs sample in one forward pass. For latency-sensitive applications, distillation of diffusion into one-step models or returning to a GAN backbone is still the right call.

GANs are not dead. They are still the right choice when you need fast sampling, when training compute is severely constrained, or when you need the precise control over latent space that StyleGAN’s W+ provides. They are no longer the default for high-quality image generation.

Pitfalls

Loss curves are not interpretable for vanilla GANs

The discriminator loss can oscillate or plateau at any value while training is going fine, and the generator loss can rise even as samples improve. The only reliable health signals for vanilla GANs are visual sample quality and FID. WGAN-GP’s critic loss correlates with $W$ and is more useful, but still not a single number to watch.

Mode collapse can hide behind sharp samples

A mode-collapsed generator produces a small number of distinct outputs, each individually sharp and convincing. Loss can look fine. Always sample a large batch, cluster, and audit per-cluster diversity; FID catches it because the marginal distribution is wrong.

Batch size and learning-rate matter more than for supervised models

Adam with $\beta_1 = 0.5$ (not 0.9) at learning rate $2 \times 10^{-4}$ is the DCGAN default. BigGAN [9] showed that scaling batch size from 256 to 2048 dropped FID dramatically, and that the same model at small batch could be unstable. Tune these before changing the architecture.

The literal less-than-or-equal sign breaks MDX

When writing GAN math in prose (e.g. “the gradient norm is at most 1”), use words rather than the literal less-than-or-equal sequence. The MDX parser interprets the angle bracket as the start of a JSX tag and crashes the build. Inside $...$ math or fenced code blocks the symbol is fine.

Pop quiz

Check yourself

Score: 0 / 5(0 answered)

1. Fixing the generator G, the optimal discriminator D*(x) is:
2. Substituting D*(x) back into V(D, G) yields:
3. Why is the JSD problematic when supports of p_data and p_g do not overlap?
4. WGAN-GP enforces the 1-Lipschitz constraint on the critic by:
5. What does Frechet Inception Distance measure?

Interview Q&A

Why is the original GAN objective a minimax game and not a single loss?onsite

The objective is

\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))].

The two networks have opposing goals: $D$ wants $V$ large (correctly classify real and fake), $G$ wants $V$ small (push $D(G(z))$ up). There is no single scalar loss whose gradient minimizes both, because making $D$ better automatically makes the data-vs-fake classification harder for $G$ .

The optimization is a saddle-point search: at equilibrium, neither player can improve unilaterally. Unlike supervised learning, there is no Lyapunov function (a scalar that monotonically decreases), so loss curves do not directly indicate progress and convergence is not guaranteed by gradient descent. Practical recipes, alternating step counts, learning-rate decay, spectral normalization, gradient penalties, exist precisely because the saddle-point is fragile.

This shape, two networks playing against each other, is what makes GANs both powerful (no need for a likelihood; you only need to be able to discriminate) and unstable (everything in the loss landscape is moving simultaneously).

Derive the optimal discriminator D*(x) = p_data(x) / (p_data(x) + p_g(x)).research

Treat $V$ as a functional of $D$ for fixed $G$ , and use a change of variables $x = G(z)$ on the second term so both expectations are over $x$ :

V(D, G) = \int_x \bigl( p_{\text{data}}(x) \log D(x) + p_g(x) \log(1 - D(x)) \bigr)\, dx.

For each fixed $x$ where $p_{\text{data}}(x) + p_g(x) > 0$ , maximize the integrand $a \log y + b \log(1 - y)$ over $y \in [0, 1]$ with $a = p_{\text{data}}(x), b = p_g(x)$ :

\frac{d}{dy} [a \log y + b \log(1 - y)] = \frac{a}{y} - \frac{b}{1 - y} = 0 \;\Longrightarrow\; y^* = \frac{a}{a + b}.

Therefore

D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}.

The interpretation is the Bayes-optimal “real vs. fake” classifier for an equal-prior mixture, i.e. the posterior probability of the real class. Substituting $D^*$ back into $V$ and adding/subtracting $\log 2$ in each integrand gives

V(D^*, G) = -\log 4 + \mathrm{KL}(p_{\text{data}} \,\Vert\, m) + \mathrm{KL}(p_g \,\Vert\, m), \quad m = \tfrac{1}{2}(p_{\text{data}} + p_g),

which equals $2 \mathrm{JSD}(p_{\text{data}} \,\Vert\, p_g) - \log 4$ by the definition of Jensen-Shannon divergence. The global optimum is at $p_g = p_{\text{data}}$ where JSD = 0, $V(D^*, G) = -\log 4$ , and $D^* \equiv \tfrac{1}{2}$ (the random-guess baseline) [1].

Why does the JSD interpretation become problematic, and how does Wasserstein GAN fix it?onsite

When $p_{\text{data}}$ and $p_g$ have disjoint supports (which is the typical situation early in training, since $p_g$ lives on a low-dimensional generator manifold), $\mathrm{JSD}(p_{\text{data}} \,\Vert\, p_g) = \log 2$ for all such configurations. The function is constant on a large open set of generator parameters, so $\nabla_G \mathrm{JSD} = 0$ and the generator has no gradient signal even though we are far from the optimum. Empirically, this manifests as the discriminator becoming “too good” early in training and the generator gradients vanishing.

The Earth Mover (Wasserstein-1) distance fixes this:

W(p, q) = \inf_{\gamma \in \Pi(p, q)} \mathbb{E}_{(x, y) \sim \gamma}[\lVert x - y \rVert].

If $p$ is a delta at $0$ and $q$ is a delta at $\theta$ , then $W(p, q) = |\theta|$ even when $\theta \neq 0$ (smooth slope), whereas $\mathrm{JSD}(p \,\Vert\, q) = \log 2$ (constant cliff). The smooth slope is exactly the gradient signal the generator needs.

The Kantorovich-Rubinstein dual rewrites $W$ as a sup over 1-Lipschitz functions:

W(p, q) = \sup_{\lVert f \rVert_L \le 1} \mathbb{E}_p[f] - \mathbb{E}_q[f],

so we can train a critic $f_w$ to estimate $W$ and use its gradient to update $G$ . Original WGAN [4] enforces the Lipschitz constraint by weight clipping; WGAN-GP [5] uses a gradient penalty $\lambda \mathbb{E}[(\lVert \nabla_{\hat x} f(\hat x) \rVert - 1)^2]$ , which trains more stably and gives a critic loss that is monotonically related to $W$ .

Derive the WGAN-GP gradient penalty and explain why it works.research

The Kantorovich-Rubinstein dual of the Earth Mover distance is

W(p, q) = \sup_{\lVert f \rVert_L \le 1} \mathbb{E}_{x \sim p}[f(x)] - \mathbb{E}_{y \sim q}[f(y)].

We want to optimize over $f_w$ parameterized as a neural net, but neural nets are not naturally Lipschitz. Original WGAN enforces the constraint by clipping all weights into $[-c, c]$ , which is crude (it shrinks the function class severely and saturates at the boundary).

WGAN-GP [5] observes that an exact 1-Lipschitz function on $\mathbb{R}^d$ has $\lVert \nabla_x f(x) \rVert_2 = 1$ almost everywhere along the optimal transport plan. So instead of constraining weights, penalize departures from unit gradient norm:

\mathcal{L}_{\text{critic}} = \underbrace{\mathbb{E}_z[f_w(G(z))] - \mathbb{E}_x[f_w(x)]}_{\text{Wasserstein estimate (negated)}} + \lambda \, \mathbb{E}_{\hat x}\!\left[ ( \lVert \nabla_{\hat x} f_w(\hat x) \rVert_2 - 1)^2 \right].

The penalty is evaluated at $\hat x = \alpha x + (1 - \alpha) G(z)$ for $\alpha \sim U[0, 1]$ and a real $x$ paired with a generated $G(z)$ . This samples points along straight lines between real and fake batches, which is where the gradient norm constraint matters most for the dual problem (the transport plan moves mass along these lines).

Why this works:

Soft constraint preserves capacity. Unlike clipping, the network can use its full weight space; the penalty just pushes the gradient norm toward 1.
Two-sided penalty hits unit norm. The squared form penalizes both larger-than-1 (Lipschitz violation) and smaller-than-1 (under-utilization) gradient norms. An exact 1-Lipschitz function has unit gradient norm a.e.
Empirically stable across architectures. The original WGAN clipping was sensitive to the clip value and to architecture choices (BatchNorm interacted badly). The gradient penalty needs no per-architecture tuning; $\lambda = 10$ works essentially everywhere.
Critic loss correlates with $W$ . Because the network approximates a 1-Lipschitz function, the inner expectation is a tight estimate of $W(p_{\text{data}}, p_g)$ , so the critic loss curve is a useful debugging signal.

A subtle point: because BatchNorm makes the loss dependent on other examples in the batch, WGAN-GP disables BatchNorm in the critic and uses LayerNorm or no normalization there.

What is Frechet Inception Distance, and why is it the standard GAN metric?onsite

FID [11] compares the distribution of real images to the distribution of generated images in the feature space of a fixed pretrained Inception-v3 network. Concretely:

Run a large batch of real images and a large batch of generated images through the Inception-v3 pool-3 layer to get 2048-dim feature vectors.
Fit a Gaussian to each set of features: $(\mu_r, \Sigma_r)$ for real, $(\mu_g, \Sigma_g)$ for generated.
Report the closed-form 2-Wasserstein distance (also called Frechet distance) between the two Gaussians:

\mathrm{FID} = \lVert \mu_r - \mu_g \rVert_2^2 + \mathrm{tr}\!\left(\Sigma_r + \Sigma_g - 2 (\Sigma_r \Sigma_g)^{1/2}\right).

Lower is better; FID = 0 means the feature distributions are identical Gaussians.

Why it is the standard:

Captures quality and diversity. A mode-collapsed generator has the wrong $\Sigma_g$ (typically degenerate or compressed), so FID rises even if individual samples look sharp. Inception Score, by contrast, can be high for a few cherry-picked classes and miss collapse.
Stable across model checkpoints. A few thousand samples are enough to estimate FID well; the pretrained Inception is fixed across all experiments so numbers are comparable.
Strongly correlated with human judgments of quality on natural images.

Caveats:

FID depends on the pretrained Inception, which was trained on ImageNet. On out-of-distribution domains (medical images, art, faces) the comparison is biased; FID is most meaningful when both real and fake look like ImageNet content.
FID is biased downward as the sample size grows; you need to use the same large sample size (typically 50k) when comparing across papers.
For text-conditional generation, FID against the conditioned subset (CLIP-FID, FID-CLIP) is more meaningful than overall FID.

In practice, FID is reported alongside Inception Score and (for class-conditional models) per-class FID, and it has been the headline metric for GAN comparisons since 2017.

Build it yourself

A 2-3 day project that takes a GAN from textbook to working samples:

Implement the DCGAN above on CIFAR-10 at 32x32 (drop the last upsampling layer in $G$ and the first downsampling layer in $D$ ). Train for 50k generator steps with Adam(lr=2e-4, betas=(0.5, 0.999)). Save sample grids every 1000 steps. Expect FID around 35-50.
Switch the loss to WGAN-GP. Disable BatchNorm in the critic (keep it in $G$ ), set $\lambda = 10$ , do 5 critic steps per generator step. Expect FID to drop and training stability to improve.
Add conditional generation: concatenate a one-hot class label to the noise input for $G$ and to the input feature map for $D$ (replicated spatially). Sample class-conditioned grids and check that horses look like horses.
Write an FID evaluation script using a pretrained Inception-v3. Report FID after every 5000 steps; this is your training health signal.
Stretch: implement spectral normalization in the critic and compare against gradient penalty on training stability and FID.
Stretch: set up StyleGAN2 (use the official NVIDIA reference repo) on FFHQ-256. Reproduce one of the truncation-tradeoff plots from the paper.

References

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Nets. NeurIPS 2014 (arXiv preprint 2014). arXiv:1406.2661
Mirza, M. and Osindero, S. (2014). Conditional Generative Adversarial Nets. arXiv preprint. arXiv:1411.1784
Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. ICLR 2016 (arXiv preprint 2015). arXiv:1511.06434
Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein GAN. ICML 2017. arXiv:1701.07875
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. (2017). Improved Training of Wasserstein GANs. NeurIPS 2017. arXiv:1704.00028
Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2017). Progressive Growing of GANs for Improved Quality, Stability, and Variation. ICLR 2018 (arXiv preprint 2017). arXiv:1710.10196
Karras, T., Laine, S., and Aila, T. (2019). A Style-Based Generator Architecture for Generative Adversarial Networks. CVPR 2019. arXiv:1812.04948
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2020). Analyzing and Improving the Image Quality of StyleGAN. CVPR 2020. arXiv:1912.04958
Brock, A., Donahue, J., and Simonyan, K. (2018). Large Scale GAN Training for High Fidelity Natural Image Synthesis. ICLR 2019 (arXiv preprint 2018). arXiv:1809.11096
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV 2017. arXiv:1703.10593
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS 2017. arXiv:1706.08500
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org

Interactive Demo

Discriminator Dominating

Epoch: 0G Loss: 2.640D Loss: 0.200

Training Loss Curves

Data Distributions

Real data (fixed)Generated data (converging)

Tip: Click Run 100 Epochs and watch the generated distribution converge toward the real distribution

Autoencoders & VAEs Practical Projects