Convolutional Neural Networks

Background

LeCun et al. (1989) introduced convolutional networks; AlexNet (2012) triggered the deep learning revolution on ImageNet
Motivated by the human visual cortex: V1 neurons respond to oriented edges, V2 to curves, higher areas to objects
Convolutions exploit translational invariance: a cat detector should work anywhere in the image

The Concept

CNNs apply learned filters (kernels) that slide across an image, detecting features regardless of where they appear. Pooling layers reduce spatial resolution while retaining the most important activations. Stacked conv+pool layers build a hierarchy from edges to textures to object parts to whole objects.

How it Links

ViT (Vision Transformer) replaced convolutions with patch-based self-attention for large-scale vision
Depthwise separable convolutions (MobileNet) and dilated convolutions connect to efficient architecture design
Feature maps from pretrained CNNs are used as visual embeddings in multimodal models

Concrete Example

ResNet-50 classifying a dog photo: layer 1 detects horizontal/vertical/diagonal edges. Layer 10 detects fur textures. Layer 40 detects ear shapes and snout profiles. Layer 50 combines them to classify the breed. Transfer learning reuses layers 1-40 for a new task like tumor detection.

Prerequisites

Multi-Layer Perceptron, Backpropagation & Optimizers, Regularization in Deep Nets

Learning outcomes

Trace the lineage from Hubel & Wiesel's simple/complex cells through the Neocognitron, LeNet, and AlexNet to modern ConvNeXt and ViT.
Compute the output spatial size and receptive field of an arbitrary convolutional stack from kernel size, stride, and padding.
Prove that convolution is translation-equivariant and explain why pooling adds local translation invariance.
Explain the residual-block gradient-flow argument: why y = F(x) + x lets gradients reach the early layers of very deep networks.
Implement 2D convolution from scratch in NumPy and write the equivalent in PyTorch and Keras.

TL;DR

A convolutional neural network slides a small learned kernel over a (typically image) input, sharing weights across all spatial positions. This enforces translation equivariance and slashes parameter count compared with a fully-connected MLP. The biological motivation comes from Hubel & Wiesel’s 1962 finding [1] of simple and complex cells in cat visual cortex; Fukushima’s Neocognitron [2] first turned this into an architecture, and LeNet-5 [3] made it work on MNIST. AlexNet [4] in 2012 was the breakout moment with ReLU + dropout + GPUs, kicking off the deep-learning era surveyed in [14]; VGG [5] showed depth via stacks of 3x3 convs; GoogLeNet [6] introduced 1x1 bottlenecks and Inception modules; ResNet [7] used skip connections to train 100+ layer networks; DenseNet [9] took the idea further by concatenating all previous feature maps; EfficientNet [10] formalized compound scaling; ViT [11] showed transformers can match CNNs on enough data; ConvNeXt [12] closed most of that gap by porting transformer design choices back into a CNN.

Intuition

Imagine recognizing a digit on a 28x28 image with an MLP. The first layer of an MLP is a fully connected matrix $\mathbb{R}^{784} \to \mathbb{R}^h$ , which means the weight that detects “horizontal stroke at the top of the image” is a different parameter from the weight that detects the same stroke shifted three pixels right. The MLP has to learn both from scratch, and it has no built-in notion that the digit might be slightly off-center.

A convolutional layer flips this around. It learns one small kernel (say 3x3) and slides it over every position, sharing the same weights everywhere. If the same stroke shows up shifted, the same kernel still fires; the layer is translation-equivariant by construction. The cost is one little matrix’s worth of parameters per output channel, instead of one big dense matrix per pixel.

That is the entire engineering case for CNNs in vision: parameter sharing turns “where in the image” from a learnable nuisance into a structural prior.

The biological story (and why it is roughly correct)

Hubel & Wiesel [1] recorded from cells in cat primary visual cortex (V1) and found two populations:

Simple cells respond to oriented edges at a specific retinal location. They are well modelled as a small filter convolved with the local image.
Complex cells respond to the same oriented edges but tolerate small shifts. They behave like a max or sum over neighboring simple cells.

Fukushima’s Neocognitron [2] turned this directly into S-cells (simple, convolutional) and C-cells (complex, pooling), trained without backprop. LeCun’s LeNet [3] kept the same simple-then-complex motif but trained it end-to-end with backprop. Modern CNNs are still recognizably that architecture: stacks of (conv, non-linearity, optional pool) blocks. The biological correspondence is loose past V1, but as an inductive bias for vision it has aged extremely well.

Math

Convolution vs cross-correlation, output size, receptive field

What “convolution” actually means in deep learning

The mathematical convolution of a 1D signal $f$ with a kernel $g$ is

(f * g)[n] = \sum_{m} f[m]\, g[n - m].

The kernel is flipped before sliding. In deep learning libraries (PyTorch’s nn.Conv2d, TensorFlow’s tf.nn.conv2d, NumPy’s scipy.signal.correlate2d) the operator that is actually applied is cross-correlation, with no flip:

(f \star g)[n] = \sum_{m} f[m]\, g[n + m].

For a learned kernel this distinction is irrelevant: gradient descent will learn whatever flipped or unflipped kernel the data demands. We use the word “convolution” colloquially to mean cross-correlation. The 2D version with stride 1 and no padding, for an input $X \in \mathbb{R}^{H \times W}$ and kernel $K \in \mathbb{R}^{k \times k}$ , is

Y[i, j] = \sum_{a=0}^{k-1} \sum_{b=0}^{k-1} X[i + a,\, j + b]\, K[a, b].

With multiple input channels $C_{\text{in}}$ and output channels $C_{\text{out}}$ , the kernel becomes a 4D tensor $K \in \mathbb{R}^{C_{\text{out}} \times C_{\text{in}} \times k \times k}$ and we sum over the input-channel axis as well.

Output size formula

For input width $W$ , kernel size $K$ , padding $P$ on each side, and stride $S$ , the output width is

W_{\text{out}} = \left\lfloor \frac{W + 2P - K}{S} \right\rfloor + 1.

Same formula in the height dimension. Three useful corollaries:

Same padding (preserve shape) at stride 1: $P = (K - 1) / 2$ , which works cleanly for odd $K$ . This is why kernel sizes are odd (1, 3, 5, 7) almost everywhere in practice.
Stride-2 conv halves the spatial size (with $P = 0$ this is approximately $W / 2$ ).
No padding shrinks by $K - 1$ pixels per side per layer, so a 3x3 stack at zero padding loses 2 pixels per layer.

Receptive field of a stacked architecture

The receptive field of a unit at layer $l$ is the set of input pixels that can affect its value. For a stack of layers with kernel sizes $K_1, K_2, \dots, K_L$ and strides $S_1, S_2, \dots, S_L$ , the receptive field grows by the recurrence

R_{l+1} = R_l + (K_{l+1} - 1) \prod_{i = 1}^{l} S_i, \qquad R_0 = 1.

The product factor encodes the fact that downstream of a stride- $S$ layer, every kernel step at the next layer corresponds to $S$ steps in the original input. Two examples:

Five 3x3 conv layers, stride 1. $R_1 = 3,\, R_2 = 5,\, R_3 = 7,\, R_4 = 9,\, R_5 = 11$ . Linear growth, $1 + 2L$ .
Same five 3x3 convs but stride 2 each. $R_1 = 3,\, R_2 = 3 + 2 \cdot 2 = 7,\, R_3 = 7 + 2 \cdot 4 = 15,\, R_4 = 31,\, R_5 = 63$ . Geometric growth.

This is why striding (or pooling) is essential: linear receptive-field growth needs $O(H)$ layers to cover an image, geometric growth needs $O(\log H)$ .

Translation equivariance, pooling, and translation invariance

Equivariance vs invariance

A function $f$ is equivariant under a transformation $T$ if $f(T x) = T f(x)$ : applying $T$ to the input shows up as the same $T$ on the output. It is invariant if $f(T x) = f(x)$ : the output does not change at all.

For 2D translation $T_\tau(x)[i, j] = x[i - \tau_1, j - \tau_2]$ :

A convolutional layer is equivariant to translation. Translating the input by $\tau$ translates the output feature map by $\tau$ (modulo boundary effects). The proof drops out of the definition: shift the input, shift the index in the sum.
A global pooling layer (max or average over the entire spatial map) is invariant to translation: translating the input does not change the global statistic. Local pooling (e.g. 2x2 max pool) is approximately invariant within its receptive field and equivariant across larger shifts.

Together, the (conv + local pool) motif gives equivariance at the small scale and invariance at the large scale, which matches the way object identity is preserved under small camera shifts but the location of the object is preserved in the feature map.

Proof of translation equivariance

Let $Y = X \star K$ at stride 1 with no padding, so $Y[i, j] = \sum_{a, b} X[i + a, j + b] K[a, b]$ . Apply a translation $T_\tau$ to the input: $X'[i, j] = X[i - \tau_1, j - \tau_2]$ . Then

Y'[i, j] = \sum_{a, b} X'[i + a,\, j + b]\, K[a, b] = \sum_{a, b} X[(i - \tau_1) + a,\, (j - \tau_2) + b]\, K[a, b] = Y[i - \tau_1,\, j - \tau_2].

So $Y' = T_\tau Y$ : shifting the input shifts the output by the same amount. Convolution commutes with translation. Without weight sharing this would not be true. Note that strided convolution preserves equivariance only up to the stride (the output is downsampled), which is part of why we still call it “translation-equivariant.”

Pooling

Two flavors:

Max pooling: $Y[i, j] = \max\{ X[Si + a, Sj + b] : 0 \le a, b < k \}$ . Picks the strongest activation in the window. Well-defined gradient (routes to the argmax), introduces some local invariance.
Average pooling: $Y[i, j] = \frac{1}{k^2} \sum_{a, b} X[Si + a, Sj + b]$ . Smoother and differentiable everywhere.

Modern architectures often replace pooling with strided convolutions (springer net, ResNet’s stride-2 conv blocks) because a strided conv is itself a learned downsampling operator and gives the same spatial reduction as pooling without throwing away information.

Architecture milestones

Most of the engineering of CNNs is in how layers are stacked. The chronology is the simplest way to internalize the design space:

1962Hubel & Wiesel: simple/complex cellsbiology
Cat V1 has direction-selective simple cells and translation-tolerant complex cells. The biological prior for CNNs.
1980Neocognitron (Fukushima)architecture
Stacked S-cells (simple, conv-like) and C-cells (complex, pooling-like). No backprop yet.
1998LeNet-5 (LeCun et al.)architecture
Conv + subsample + conv + subsample + FC, trained end-to-end with backprop on MNIST. The blueprint.
2012AlexNet (Krizhevsky, Sutskever, Hinton)breakthrough
ReLU activations, dropout, two GPUs. Cut ImageNet top-5 error from 26% to 15%. Triggered the deep-learning era.
2014VGG (Simonyan, Zisserman)architecture
Just stack 3x3 convs many times deep. Showed that depth alone, with no fancy tricks, drives accuracy.
2014GoogLeNet / Inception (Szegedy et al.)architecture
Inception module: parallel 1x1, 3x3, 5x5, and pool branches concatenated, with 1x1 convs as channel bottlenecks.
2015ResNet (He et al.)breakthrough
Skip connections y = F(x) + x make 100+ layer networks trainable. Won ILSVRC 2015 with a 152-layer net.
2017DenseNet (Huang et al.)architecture
Each layer concatenates all previous feature maps. Strong feature reuse, fewer parameters.
2019EfficientNet (Tan, Le)architecture
Compound scaling: jointly scale depth, width, and resolution along a single coefficient.
2020Vision Transformer / ViT (Dosovitskiy et al.)paradigm shift
A pure transformer on 16x16 image patches matches the best CNNs given enough data (JFT-300M).
2022ConvNeXt (Liu et al.)architecture
Port modern transformer design choices (LayerNorm, GELU, large kernels) into a CNN; close most of the ViT gap.

A side-by-side of the famous models on ImageNet:

Model	Year	Depth	Params	Key idea	ImageNet top-1
LeNet-5	1998	7	~60K	5x5 conv + avg-pool, sigmoid	n/a (MNIST)
AlexNet	2012	8	~60M	ReLU + dropout + 2 GPUs + LRN	~63%
VGG-16	2014	16	~138M	stacks of 3x3 conv, no fancy blocks	~71%
GoogLeNet (Inception-v1)	2014	22	~7M	inception modules, 1x1 bottlenecks	~69%
ResNet-50	2015	50	~25M	skip connections y = F(x) + x	~76%
ResNet-152	2015	152	~60M	same idea, much deeper	~78%
EfficientNet-B7	2019	~50 effective	~66M	compound scaling	~84%
ViT-L/16	2020	24 transformer blocks	~307M	transformer on 16x16 patches	~85% (with JFT pretrain)
ConvNeXt-L	2022	36 conv stages	~198M	modernized CNN matches ViT	~85%

The ResNet identity-mapping argument

ResNet [7] was the first paper that made networks past 30-50 layers trainable in practice. Its key building block is the residual block:

y = F(x; W) + x.

Here $F$ is a small stack of conv layers, and $x$ is added back as a skip connection (identity shortcut). In the follow-up paper [8] the authors made the identity nature explicit: any layer from $L_1$ to $L_2$ can be written as

x_{L_2} = x_{L_1} + \sum_{l = L_1}^{L_2 - 1} F(x_l;\, W_l).

Differentiating gives the gradient flow

\frac{\partial \mathcal{L}}{\partial x_{L_1}} = \frac{\partial \mathcal{L}}{\partial x_{L_2}}\!\left( 1 + \frac{\partial}{\partial x_{L_1}} \sum_{l = L_1}^{L_2 - 1} F(x_l; W_l) \right).

The constant 1 is the punchline: even if every $F$ has a tiny gradient, the gradient of the loss with respect to early activations cannot be smaller than $\partial \mathcal{L} / \partial x_{L_2}$ . Skip connections give gradients an unobstructed path, which is what makes very deep nets trainable. Empirically a 152-layer ResNet trains with the same recipe as a 34-layer one; without the skip, the 152-layer plain net trains worse than the 34-layer one (the famous “degradation” plot in [7]).

Implementations

Three idiomatic implementations of the same idea: a from-scratch NumPy 2D convolution function, a PyTorch LeNet-style CNN, and a Keras Sequential equivalent.

python

import numpy as np

def conv2d(x, kernel, stride=1, padding=0):
  """Naive 2D cross-correlation. x: (H, W). kernel: (kH, kW)."""
  H, W = x.shape
  kH, kW = kernel.shape
  if padding > 0:
      x = np.pad(x, padding, mode='constant', constant_values=0.0)
      H, W = x.shape
  out_H = (H - kH) // stride + 1
  out_W = (W - kW) // stride + 1
  out = np.zeros((out_H, out_W), dtype=x.dtype)
  for i in range(out_H):
      for j in range(out_W):
          patch = x[i*stride : i*stride + kH, j*stride : j*stride + kW]
          out[i, j] = (patch * kernel).sum()
  return out

# Sobel-x edge detector on a synthetic 5x5 image.
img = np.array([
  [0, 0, 1, 0, 0],
  [0, 0, 1, 0, 0],
  [0, 0, 1, 0, 0],
  [0, 0, 1, 0, 0],
  [0, 0, 1, 0, 0],
], dtype=float)

sobel_x = np.array([
  [-1, 0, 1],
  [-2, 0, 2],
  [-1, 0, 1],
], dtype=float)

print(conv2d(img, sobel_x, stride=1, padding=0))

About 20 lines including padding and stride. The two for-loops are O(out_H * out_W * kH * kW); production code reshapes via im2col + GEMM.

Interactive: 2D convolution

The visual demo first: edit the kernel below (try the Identity, Edge, and Blur presets) and watch the 7x7 output feature map update live. The input is a fixed 9x9 toy image; the kernel is a 3x3 cross-correlation kernel applied at stride 1 with no padding.

2D convolution (stride 1, no padding) on a 9x9 toy image

input (9 x 9)

kernel (3 x 3, editable)

output (7 x 7)

Cell colors map to the value (clamped to [-2, 2]); positive values lean red and negative values lean blue. The output map is computed live from the editable kernel.

Then the runnable from-scratch verification, which applies a Sobel-x edge detector to a small synthetic image:

Idle (packages: numpy)

(no output yet)

Real-world uses

CNNs were the dominant computer-vision architecture from 2012 to roughly 2020 and are still ubiquitous in production:

Domain	Use	Why a CNN works
ImageNet classification	Whole-image label	Translation equivariance + hierarchical features match natural images.
Object detection	Faster R-CNN, RetinaNet, YOLO families	CNN backbone produces a dense feature map; detection head reads boxes off it.
Semantic segmentation	U-Net, DeepLab	Encoder-decoder CNN with skip connections; per-pixel classification.
Medical imaging	X-ray, CT, MRI; diabetic-retinopathy screening	Translation equivariance + small dataset friendliness; transfer learning from ImageNet pretraining.
Satellite imagery	Land cover, change detection	Same shift-equivariance argument; CNNs are also translation-equivariant in the spatial dimensions of remote-sensing imagery.
Audio (spectrograms)	Music tagging, keyword spotting, speech	A spectrogram is an image; a 2D CNN over time x frequency works well.

Pitfalls

Stride and padding interact off-by-one

The output-size formula $\lfloor (W + 2P - K) / S \rfloor + 1$ has a floor. With even $K$ or non-default $S$ , the floor can drop the rightmost column or bottom row, leading to “where did my pixel go” bugs. Stick to odd kernels (3, 5, 7) with same padding, or check the exact output shape before wiring into the next layer. PyTorch and Keras silently use this formula; only the values $K$ , $P$ , $S$ that satisfy $(W + 2P - K) \mod S = 0$ avoid information loss at the boundary.

Batch norm needs a reasonable batch size

The per-batch statistics that batch norm computes are only stable at batch size at least 16 to 32 per device. With micro-batches (semantic segmentation, video, very large models split across many GPUs) the statistics are noisy and training degrades. Use group norm or layer norm instead in those settings; see Regularization in Deep Nets.

Transfer learning is the default, not from-scratch training

Outside very large training datasets (ImageNet-22k, JFT-300M, LAION) you almost never train a CNN from random init. The standard recipe is: take a pretrained ImageNet (or DINO, or CLIP) backbone, replace the head, and fine-tune. From-scratch training on ten thousand or fewer images usually overfits before it learns useful features.

Pooling has fallen out of fashion

Modern architectures (ResNet from 2015 onward, ConvNeXt, ViT) replace explicit pooling with stride-2 convolutions or patch tokenization. The empirical case: a stride-2 conv is a learned downsampling operator that generalizes max-pool (which throws away $k^2 - 1$ activations per window). Global average pooling at the very end of the network, before the final linear classifier, has stuck around because it gives translation invariance for free.

Pop quiz

Check yourself

Score: 0 / 5(0 answered)

1. A 2D convolutional layer with input width 32, kernel size 5, padding 2, and stride 2 produces output width:
2. Why is convolution translation-equivariant?
3. Why does a residual block y = F(x) + x help with very deep networks?
4. What is the receptive field at layer 3 of a stack of three 3x3 conv layers, all stride 1?
5. What does a 1x1 convolution do?

Interview Q&A

Derive the receptive-field recurrence R_{l+1} = R_l + (K_{l+1} - 1) * prod over i of S_i for i at most l.onsite

Define $R_l$ as the receptive field (in input pixels) of one unit at layer $l$ . At layer 0 (the input), $R_0 = 1$ (a single pixel sees only itself).

A layer with kernel size $K_{l+1}$ applied at stride $1$ on the previous feature map enlarges the receptive field by $K_{l+1} - 1$ , because each unit in the new feature map looks at $K_{l+1}$ adjacent units below it, and adjacent units have receptive fields shifted by exactly 1 input pixel. So with all strides 1:

R_{l+1} = R_l + (K_{l+1} - 1).

When we add strides, the spacing between adjacent units in the previous feature map (measured in input pixels) is no longer 1 but $\prod_{i \le l} S_i$ : a unit at $(i, j)$ in layer $l$ corresponds to input pixel $(i \cdot \prod S_i, j \cdot \prod S_i)$ . So when layer $l+1$ steps to its neighboring unit, it skips $\prod_{i \le l} S_i$ input pixels. The recurrence becomes

R_{l+1} = R_l + (K_{l+1} - 1) \prod_{i \le l} S_i.

This is why striding (or pooling) is essential for covering large images with few layers: the effective stride grows multiplicatively. A network with $L$ stride-2 layers has effective stride $2^L$ , so the receptive field grows as $\sum_l 2^l$ which is geometric in $L$ [13].

Prove that 2D convolution is translation-equivariant.phone

Let $Y = X \star K$ at stride 1 (no padding) so $Y[i, j] = \sum_{a, b} X[i + a, j + b]\, K[a, b]$ . Apply translation $T_\tau$ to the input: $X'[i, j] = X[i - \tau_1, j - \tau_2]$ . Then

Y'[i, j] = \sum_{a, b} X'[i + a, j + b]\, K[a, b] = \sum_{a, b} X[i + a - \tau_1, j + b - \tau_2]\, K[a, b] = Y[i - \tau_1, j - \tau_2].

Hence $Y' = T_\tau Y$ . The output is translated by exactly the same vector as the input. The argument relies on the kernel being independent of position (weight sharing), which is why a fully-connected layer applied to flattened pixels is not translation-equivariant.

Note that strided convolution preserves equivariance only up to translations that are multiples of the stride; this is part of the reason all-convolutional networks tend to keep some pooling or include shift-equivariant downsampling such as blurpool.

Why does the ResNet skip connection help training, mathematically?onsite

A residual block is $x_{l+1} = x_l + F(x_l; W_l)$ . By induction, the activation at any later layer $L_2$ is

x_{L_2} = x_{L_1} + \sum_{l = L_1}^{L_2 - 1} F(x_l;\, W_l).

Differentiating with respect to an early activation $x_{L_1}$ :

\frac{\partial \mathcal{L}}{\partial x_{L_1}} = \frac{\partial \mathcal{L}}{\partial x_{L_2}} \cdot \left( I + \frac{\partial}{\partial x_{L_1}} \sum_{l} F(x_l; W_l) \right).

The first term in the parenthesis is the identity $I$ , regardless of how many residual blocks lie between $L_1$ and $L_2$ and regardless of how small the Jacobians of the $F$ blocks are. So the gradient signal from the loss reaches $x_{L_1}$ unattenuated, plus a (possibly small) contribution from the residual path.

In a plain feed-forward stack without skips, the chain rule multiplies all the per-layer Jacobians together. If those Jacobians have spectral norm even a bit below 1, the product shrinks geometrically with depth and gradients vanish. Skip connections are the structural fix, and they are the reason 100+ layer networks (ResNet-152, ResNet-1000) are trainable while the equivalent plain stack is not [7][8].

What does a 1x1 convolution do, and why is it useful?phone

A 1x1 conv with $C_{\text{in}}$ input channels and $C_{\text{out}}$ output channels is a per-pixel linear map $\mathbb{R}^{C_{\text{in}}} \to \mathbb{R}^{C_{\text{out}}}$ , applied independently at each spatial location. There is no spatial mixing; it only mixes across channels.

Two heavy uses:

Channel bottleneck (Inception, ResNet). Reduce $C_{\text{in}}$ before an expensive $3 \times 3$ conv, then expand back. This cuts FLOPs from $9 C_{\text{in}} C_{\text{out}}$ per spatial position to $C_{\text{in}} c + 9 c c + c C_{\text{out}}$ for a small bottleneck width $c$ . ResNet’s bottleneck block does exactly this.
Network-in-network / pointwise blocks. A stack of 1x1 convs with non-linearities is a per-pixel MLP. ConvNeXt and MobileNetV2’s inverted-residual blocks rely heavily on this pattern.

In a transformer, the attention output is followed by a two-layer MLP applied independently per token, which is structurally identical to two 1x1 convs over a 1D feature map [6].

Why has pooling fallen out of fashion in modern architectures?research

Three reasons that compound:

Information loss. Max pooling discards $k^2 - 1$ values per window; average pooling smooths over local detail. A stride-2 conv applies a learned linear combination of the $k^2$ inputs and is strictly more expressive (and learns whatever pooling-like behavior the data prefers).
Differentiability. Max pool’s gradient routes only to the argmax; under heavy training noise this is unstable. Stride-2 conv’s gradient routes everywhere with smooth weights.
Modern architectures (ResNet, ConvNeXt, ViT) downsample with stride-2 convs or with patch tokenization (ViT splits an image into 16x16 patches once, then no further downsampling until the head). Pooling is mostly relegated to the very end as global-average-pool, which gives translation invariance and a fixed-size feature regardless of input resolution [7][11][12].

The argument is empirical, not theoretical: networks that replace pool with stride-conv generalize at least as well and often better, with no loss of efficiency once the stride is folded into the conv.

Build it yourself

A weekend project that takes a CNN from textbook to practical:

Implement the LeNet-5 above in PyTorch and reach 99% on MNIST.
Swap to CIFAR-10 and add data augmentation (random crops, horizontal flips). Pure LeNet caps around 75%; add batch norm to push past 80%.
Replace the MLP head with global average pooling, add three residual blocks, and reach 90%+.
Stretch: load a pretrained ResNet-50 from torchvision, fine-tune the last block on a 100-class subset of ImageNet, and watch fine-tuning beat from-scratch by 10-20 absolute points.
Compare against a small ViT (timm.create_model('vit_tiny_patch16_224', pretrained=True)) on the same task and read off the data-efficiency vs in-domain-pretraining trade-off.

References

Hubel, D. H. and Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. Journal of Physiology 160(1), 106-154. doi:10.1113/jphysiol.1962.sp006837
Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics 36(4), 193-202. doi:10.1007/BF00344251
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278-2324. doi:10.1109/5.726791
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. NeurIPS 2012. https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
Simonyan, K. and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. ICLR 2015 (arXiv preprint 2014). arXiv:1409.1556
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2014). Going Deeper with Convolutions. CVPR 2015 (arXiv preprint 2014). arXiv:1409.4842
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep Residual Learning for Image Recognition. CVPR 2016 (arXiv preprint 2015). arXiv:1512.03385
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Identity Mappings in Deep Residual Networks. ECCV 2016. arXiv:1603.05027
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017). Densely Connected Convolutional Networks. CVPR 2017. arXiv:1608.06993
Tan, M. and Le, Q. V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019. arXiv:1905.11946
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR 2021 (arXiv preprint 2020). arXiv:2010.11929
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022). A ConvNet for the 2020s. CVPR 2022. arXiv:2201.03545
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org
LeCun, Y., Bengio, Y., and Hinton, G. (2015). Deep learning. Nature 521, 436-444. doi:10.1038/nature14539

Regularization in Deep Nets RNNs, LSTMs, GRUs