Regularization in Deep Nets

TL;DR

Intuition

A modern deep network has tens of millions to billions of parameters and is trained on a fraction of the data needed to constrain them all. Without help, gradient descent will find a configuration that fits the training set perfectly and generalizes badly. Regularization is the umbrella term for everything that biases optimization toward simpler, smoother, or more invariant functions.

The techniques in this section split into three broad families:

• Parameter penalties. Add a term to the loss that shrinks parameters (L1, L2, weight decay).
• Stochastic noise injection. Randomly perturb activations, layers, or labels at training time so the model cannot memorize specific patterns (dropout, stochastic depth, label smoothing, MixUp/CutMix, RandAugment).
• Normalization. Standardize internal activations to keep their distribution stable across the network and across training, removing brittle dependencies on initialization and learning-rate (batch norm, layer norm, group norm).

Almost every production training recipe combines one technique from each family.

Weight decay

The plain L2 regularizer adds $\tfrac{\lambda}{2}\lVert \theta \rVert_2^2$ to the loss. The gradient becomes $\nabla \mathcal{L} + \lambda \theta$, and a plain SGD step is

This is decay in the literal sense: a uniform multiplicative shrinkage of $\theta$. Under SGD this is identical to L2 regularization.

Dropout

Dropout [1] independently sets each activation to zero with probability $p$ at training time. To keep the expected activation magnitude unchanged, the kept activations are scaled by $1/(1-p)$ at training time (this is “inverted dropout”; original dropout scaled by $1-p$ at inference, but every modern framework uses inverted dropout):

At inference time, dropout is off: $h_i^{\text{eval}} = h_i$.

The implicit ensemble

A network with $n$ dropout-able units defines $2^n$ possible sub-networks (each of the $n$ units kept or dropped). Each minibatch trains one of these sub-networks, sampled uniformly. At inference, using the full network with all activations multiplied by $1/(1-p)$ approximates the geometric mean of the predictions of all $2^n$ sub-networks. So dropout is implicitly training an ensemble of $2^n$ models and averaging them at inference, all at the cost of training a single model.

Where to put it

Typical placements:

• MLPs and FFN blocks of transformers: between hidden layers, $p \in [0.1, 0.5]$.
• Just before the final logits: a smaller $p$ to prevent overconfidence on the output.
• Within attention: attention dropout (drop attention weights) and residual dropout (drop the output of each sub-layer) are standard in transformer training.

Do not stack dropout with strong batch norm in the same path: BN’s stats interact badly with dropout’s noise; the canonical fix is “dropout after BN” or “no dropout, use BN only.”

Batch normalization

Batch norm [2] standardizes each pre-activation along the batch dimension. For a tensor of pre-activations $z \in \mathbb{R}^{B \times C}$ (batch size $B$, channels $C$), compute per-channel statistics across the batch:

and normalize, then apply learned scale $\gamma_c$ and shift $\beta_c$:

At training time, $\mu, \sigma$ are computed per minibatch. The layer also maintains exponentially-moving-average running estimates $\bar{\mu}, \bar{\sigma}$ for inference time, when the input may be a single example.

Layer norm and group norm

Batch norm fails in two settings: (a) batch size is very small (e.g. semantic segmentation where each batch is a few images), so the per-channel statistics are noisy; (b) sequence-to-sequence models where the batch dimension is shuffled with the time dimension and “per channel across batch” is no longer meaningful.

Layer norm [3]: normalize across the feature dimension per sample. For activations $h \in \mathbb{R}^{B \times d}$,

then $\hat{h}_{i,j} = \gamma_j (h_{i,j} - \mu_i) / \sqrt{\sigma_i^2 + \epsilon} + \beta_j$. Training and eval modes are identical (no running stats needed). Layer norm is the default in transformers (BERT, GPT, ViT), where the batch dimension is heterogeneous and the feature dimension is the natural axis to normalize.

Group norm [4]: split the channel dimension into $G$ groups and normalize per (sample, group). For $C$ channels and group size $C/G$, this interpolates between layer norm ($G = 1$, normalize over all channels) and instance norm ($G = C$, normalize each channel independently). Group norm is the standard fix for CV with very small batch size (semantic segmentation, video, medical imaging) where batch norm’s per-batch statistics are too noisy.

Other regularization tools

Early stopping

Track validation loss every epoch (or every $k$ steps). Save the best-validation-loss checkpoint. When validation loss stops improving for $P$ epochs (the “patience”), restore the best checkpoint and stop training. Equivalent to a soft cap on effective model complexity: by stopping before the train loss reaches its minimum, you stop in the underfit-overfit transition rather than the overfit regime.

Label smoothing

Replace one-hot labels with a soft target:

where $K$ is the number of classes and $\alpha \in [0.05, 0.2]$ is typical. Introduced in Inception-v3 [6].

Data augmentation

Replace each training example with a random transformation that preserves the label.

• Standard CV augmentation. Random crop, horizontal flip, color jitter, slight rotation. The classic recipe behind ImageNet training.
• MixUp [7]. Sample two examples $(x_1, y_1)$ and $(x_2, y_2)$ and a $\lambda \sim \text{Beta}(\alpha, \alpha)$. Train on $\bigl( \lambda x_1 + (1 - \lambda) x_2,\, \lambda y_1 + (1 - \lambda) y_2 \bigr)$. Encourages the network to behave linearly in between training examples; reduces overconfidence.
• CutMix [8]. Cut a random rectangle out of one image and paste in the corresponding region from another, mixing the labels by the area ratio. Combines the regularization of MixUp with the spatial inductive bias that small image patches still mean something.
• RandAugment [9]. Sample $N$ augmentations from a fixed library (rotate, shear, color, etc.) and apply each at magnitude $M$, both treated as hyperparameters. Replaces the AutoAugment search with two scalars; nearly as good and 1000× cheaper.

Stochastic depth

Stochastic depth [10]: at each training step, randomly skip the residual computation of a fraction of the residual blocks (replace $x + f(x)$ with $x$ at probability $p_\ell$). At inference time use the full network. Like dropout but applied to entire layers; particularly useful for very deep ResNets and ViTs (e.g. ViT-Huge with stochastic depth $p = 0.4$).

Weight tying

Tie the input embedding matrix and the output softmax projection in language models [12]. Cuts parameter count by ~30M for a typical language model and improves perplexity. Standard in GPT-style models. The intuition: the input embedding maps tokens to vectors, and the output projection maps vectors back to token logits; both should encode the same word-vector geometry, so why learn them twice.

Implementations

A small CNN in PyTorch with each of the major techniques wired in. Toggle them on or off to recreate any modern training recipe.

import torch.nn as nn

# Small CNN for CIFAR-10 with: BN, dropout, GELU, weight init.
# AdamW + label smoothing + MixUp are wired in the training loop.

class Block(nn.Module):
  def __init__(self, c_in, c_out, drop=0.1):
      super().__init__()
      self.conv = nn.Conv2d(c_in, c_out, kernel_size=3, padding=1, bias=False)
      self.bn   = nn.BatchNorm2d(c_out)         # batch norm after conv
      self.act  = nn.GELU()
      self.drop = nn.Dropout2d(drop)             # spatial dropout
  def forward(self, x):
      return self.drop(self.act(self.bn(self.conv(x))))

class TinyCNN(nn.Module):
  def __init__(self, num_classes=10):
      super().__init__()
      self.features = nn.Sequential(
          Block(3, 64), Block(64, 64), nn.MaxPool2d(2),
          Block(64, 128), Block(128, 128), nn.MaxPool2d(2),
          nn.AdaptiveAvgPool2d(1), nn.Flatten(),
      )
      self.classifier = nn.Sequential(
          nn.Linear(128, 256), nn.GELU(), nn.Dropout(0.3),
          nn.Linear(256, num_classes),
      )
      for m in self.modules():
          if isinstance(m, nn.Conv2d):
              nn.init.kaiming_normal_(m.weight, nonlinearity='relu')
          elif isinstance(m, nn.Linear):
              nn.init.kaiming_normal_(m.weight, nonlinearity='relu')
              nn.init.zeros_(m.bias)
  def forward(self, x):
      return self.classifier(self.features(x))

Run a from-scratch numpy demo of dropout’s regularization effect: a small overfit-prone MLP trained twice (with and without dropout) on a tiny dataset:

(no output yet)

Interactive demo

Drag the polynomial-degree slider to watch a model fit (or overfit) noisy data. Higher-degree polynomials nail the training points but oscillate wildly between them, exactly the failure mode regularization is designed to tame. Mentally translate “polynomial degree” to “number of parameters in a deep net”: dropout, weight decay, and early stopping all push the effective capacity down without changing the architecture.

Real-world uses

Pitfalls

Pop quiz

Interview Q&A