RNNs, LSTMs, GRUs

Background

Elman (1990) introduced simple RNNs for sequence processing; LSTMs by Hochreiter & Schmidhuber (1997) solved the vanishing gradient problem in long sequences
Dominated NLP sequence-to-sequence tasks until transformers (2017)
Still used in time-series forecasting, audio modeling, and embedded systems where Transformers are too heavy

The Concept

RNNs process sequences one step at a time, maintaining a hidden state that carries information from past steps. LSTMs add gating mechanisms (input, forget, output gates) that control what information flows through the cell state, enabling long-range dependencies.

How it Links

The attention mechanism in Transformers was designed to overcome the fixed-size bottleneck of the RNN hidden state
Bidirectional RNNs and stacked LSTMs connect to the encoder-decoder architecture for seq2seq tasks
State space models (Mamba) are modern alternatives to RNNs for long-sequence modeling

Concrete Example

Machine translation with an LSTM: reading "the cat sat" word by word, the hidden state accumulates context. By the time it generates the Spanish translation, the LSTM uses the full sentence context stored in its cell state. The forget gate helps it drop the subject after outputting the verb.

Prerequisites

Multi-Layer Perceptron, Backpropagation & Optimizers, Convolutional Neural Networks

Learning outcomes

Define a vanilla RNN and derive backpropagation through time (BPTT) by unfolding across the sequence and applying standard backprop.
Explain the vanishing and exploding gradient problem as a product of T Jacobians (Bengio 1994; Pascanu 2013) and apply gradient clipping.
Derive the LSTM gates from the design objective of preserving long-range gradients via a gated additive cell-state recurrence (Hochreiter & Schmidhuber 1997).
Compare the LSTM, GRU (Cho 2014), and bidirectional/stacked variants; pick the right one for a given size and data regime.
Explain why Bahdanau attention (2014) and ultimately the Transformer displaced RNNs for sequence modelling.

TL;DR

A recurrent neural network shares one transition $f$ across time: $h_t = f(h_{t-1}, x_t)$ . BPTT [2] unfolds the recurrence and runs standard backprop, but the chain rule produces a product of $T$ Jacobians whose spectral norm controls whether gradients vanish or explode across long sequences [3][4]. LSTMs [5] fix vanishing gradients with a gated additive cell state $c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$ , so the gradient through $c$ is multiplied by the (often near-1) forget gate rather than a saturating Jacobian. GRUs [7] simplify to two gates and one state. Bidirectional and stacked variants are standard. Bahdanau attention [9] over the encoder hidden states removed the seq2seq [8] bottleneck, and the Transformer [11] displaced RNNs entirely for most sequence tasks because it is fully parallel across time and captures longer effective context.

Intuition

A feed-forward network sees one example at a time. A recurrent network sees a sequence, and at each step it carries forward a hidden state that summarizes everything it has read so far. The simplest version is

h_t = \tanh(W_h h_{t-1} + W_x x_t + b), \qquad y_t = W_y h_t,

and that single recurrence, applied at every time step, is what makes the network “recurrent.” The same parameters $(W_h, W_x, W_y)$ are reused across time, exactly the same way a CNN reuses its kernel across spatial positions. Sharing across time is the inductive bias for sequences.

The catch: training requires backpropagating through the entire chain $h_T \leftarrow h_{T-1} \leftarrow \cdots \leftarrow h_1$ , and the chain rule multiplies $T$ Jacobians of the recurrence. If those Jacobians shrink, gradients vanish past a few dozen steps; if they grow, gradients explode. LSTMs and GRUs are architectural fixes for the vanishing case; gradient clipping is the production fix for the exploding case.

Why parameter sharing across time matters

Consider modelling a 100-token sentence with an MLP that takes the whole sentence at once. You would need $\Theta(100 \cdot d \cdot h)$ parameters per layer and the model would have to re-learn what “the” looks like at every position separately. An RNN learns one $W_x$ that handles “the” wherever it appears, with no positional duplication. The price is that the model is sequential: $h_t$ depends on $h_{t-1}$ , so you cannot parallelize across time. That sequential dependency is the single biggest reason Transformers eventually displaced RNNs for large-scale language modelling [11].

Math

Vanilla RNN and backpropagation through time (BPTT)

Forward pass

For an input sequence $x_1, x_2, \dots, x_T$ and an initial hidden state $h_0$ , the vanilla RNN computes

h_t = \tanh(W_h h_{t-1} + W_x x_t + b_h), \qquad y_t = W_y h_t + b_y,

for $t = 1, \dots, T$ . Define a per-step loss $\ell_t(y_t, y_t^*)$ and a total loss $\mathcal{L} = \sum_t \ell_t$ .

Unfolding

Treat the recurrent network as a feed-forward network of depth $T$ where layer $t$ shares parameters with every other layer. This is “unfolding” the RNN through time. Then BPTT is just standard backpropagation [1] on the unfolded graph, with the constraint that the gradient w.r.t. each shared weight is the sum over time:

\frac{\partial \mathcal{L}}{\partial W_h} = \sum_{t=1}^{T} \frac{\partial \ell_t}{\partial W_h} = \sum_{t=1}^{T} \sum_{s \le t} \frac{\partial \ell_t}{\partial h_t} \cdot \frac{\partial h_t}{\partial h_s} \cdot \frac{\partial h_s}{\partial W_h}.

The product of Jacobians

The dangerous term is $\partial h_t / \partial h_s = \prod_{k = s+1}^{t} \partial h_k / \partial h_{k-1}$ , a product of $t - s$ Jacobians. For a vanilla RNN with tanh,

\frac{\partial h_k}{\partial h_{k-1}} = \mathrm{diag}(\mathrm{tanh}'(z_k))\, W_h,

where $z_k = W_h h_{k-1} + W_x x_k + b_h$ . Bound the spectral norm: $\lVert \mathrm{tanh}' \rVert_\infty \le 1$ , so $\lVert \partial h_k / \partial h_{k-1} \rVert \le \lVert W_h \rVert$ . The product of $T$ such factors satisfies

\left\lVert \prod_{k = s+1}^{t} \frac{\partial h_k}{\partial h_{k-1}} \right\rVert \le \lVert W_h \rVert^{t - s}.

Vanishing vs exploding

Two regimes [3][4]:

If the largest singular value of $W_h$ is less than 1, the product contracts geometrically to zero. The gradient signal from a loss at time $t$ never reaches inputs far in the past; the RNN cannot learn dependencies past about $1 / |\log \lVert W_h \rVert|$ time steps. This is the vanishing gradient problem.
If the largest singular value is greater than 1, the product grows geometrically. Gradients explode, weights take wild updates, training diverges. This is the exploding gradient problem.

In practice both happen on the same network, on different sequences. The standard production fix for explosion is gradient clipping [4]: scale the full gradient vector if its norm exceeds a threshold,

g \leftarrow \begin{cases} g & \text{if } \lVert g \rVert \le \tau, \\ \tau \cdot g / \lVert g \rVert & \text{otherwise.} \end{cases}

The standard fix for vanishing is to redesign the recurrence so it has a path with near-identity Jacobian; that is what LSTMs and GRUs do.

LSTM: deriving the gates from the gradient-flow objective

Design objective

We want a recurrence $c_t = g(c_{t-1}, \ldots)$ such that $\partial c_t / \partial c_{t-1}$ is close to the identity along most paths, even after many time steps. The simplest way to get this is to make the recurrence additive:

c_t = c_{t-1} + \Delta_t.

Then $\partial c_t / \partial c_{t-1} = I$ exactly, and gradients propagate through the cell state without contraction or explosion. But pure addition has no mechanism to forget old information or to control how much new information enters.

Adding gates

Replace the constants in the additive recurrence with gates in $[0, 1]^d$ that the network learns:

c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t,

where $f_t$ is the forget gate (how much of the old cell state to keep), $i_t$ is the input gate (how much new information to write), and $\tilde c_t$ is the candidate update (the actual new information). Each is a small MLP of the previous hidden state and current input:

\begin{aligned} i_t &= \sigma(W_i [h_{t-1}, x_t] + b_i), & f_t &= \sigma(W_f [h_{t-1}, x_t] + b_f), \\ o_t &= \sigma(W_o [h_{t-1}, x_t] + b_o), & \tilde c_t &= \tanh(W_c [h_{t-1}, x_t] + b_c). \end{aligned}

Finally the output gate $o_t$ controls what the cell exposes as the hidden state:

h_t = o_t \odot \tanh(c_t).

This is the LSTM [5], with the forget gate added by Gers, Schmidhuber, and Cummins [6] three years later (the original LSTM had no forget gate; without it the cell state grew unboundedly).

Gradient flow through the cell state

Differentiate the cell-state recurrence:

\frac{\partial c_t}{\partial c_{t-1}} = \mathrm{diag}(f_t) + \text{(small terms via } i_t, \tilde c_t \text{ that depend on } c_{t-1} \text{ through } h_{t-1}).

The dominant term is $\mathrm{diag}(f_t)$ . If the network learns to keep $f_t \approx 1$ on relevant dimensions, the cell-state gradient is multiplied by something close to the identity at each step, and gradients can flow through hundreds of time steps. Compare to vanilla RNN where the per-step factor is $\mathrm{diag}(\tanh'(z)) W_h$ , whose spectral norm is bounded by $\lVert W_h \rVert$ but can shrink arbitrarily through the saturating $\tanh'$ .

The hidden-state branch ( $h_t = o_t \odot \tanh c_t$ ) does saturate, but the cell state itself does not: that decoupling is the design insight. Long-range dependencies live in $c$ ; the squashed $h$ is what is exposed to the output.

Initial bias trick

In practice, initializing the forget-gate bias $b_f$ to a positive value (commonly $b_f = 1$ or $b_f = 2$ ) makes $f_t \approx \sigma(\text{positive}) \gtrsim 0.7$ at initialization, so the cell state retains most of its previous value before training begins. Without this, the LSTM starts in a regime where $f_t \approx 1/2$ and gradients still attenuate by $2^{-T}$ over $T$ steps, defeating the design [12].

GRU: a smaller LSTM with two gates

The Gated Recurrent Unit (GRU) [7] simplifies the LSTM in two ways:

Merge the cell state and hidden state. There is just $h_t$ , no separate $c_t$ .
Merge the input and forget gates into a single update gate. The forget probability is exactly $1$ minus the input probability.

The equations:

\begin{aligned} z_t &= \sigma(W_z [h_{t-1}, x_t]) \quad \text{(update gate)} \\ r_t &= \sigma(W_r [h_{t-1}, x_t]) \quad \text{(reset gate)} \\ \tilde h_t &= \tanh(W_h [r_t \odot h_{t-1},\, x_t]) \quad \text{(candidate)} \\ h_t &= (1 - z_t) \odot h_{t-1} + z_t \odot \tilde h_t. \end{aligned}

The reset gate $r_t$ controls how much of the previous hidden state contributes to the candidate (zero $r_t$ means “ignore the past, treat this token fresh”). The update gate $z_t$ controls the convex combination between the previous state and the candidate.

Parameter count: a GRU has roughly $3$ gate matrices (vs an LSTM’s $4$ ), so it is about 25% smaller than an LSTM with the same hidden size. Empirically GRU and LSTM perform within noise of each other on most tasks; GRU is preferred when parameters or compute are tight (smaller embeddings, edge devices, smaller datasets), LSTM is the safer default at large scale.

Bidirectional and stacked variants

Bidirectional: run a forward LSTM/GRU on $x_1, \dots, x_T$ and a backward one on $x_T, \dots, x_1$ in parallel, then concatenate the hidden states $[\overrightarrow{h}_t, \overleftarrow{h}_t]$ . Standard for sequence labelling (NER, POS tagging) where the entire sentence is available at inference. Not usable for streaming or autoregressive decoding.
Stacked: feed the hidden states of one RNN as inputs to another RNN above it. Stacking deepens the temporal feature hierarchy; 2-3 layers is typical, more rarely helps and adds optimization issues. Modern speech recognition pipelines (Graves et al. [10]) stacked deep bidirectional LSTMs with batch norm or layer norm between layers.

Bahdanau attention: removing the seq2seq bottleneck

The original seq2seq architecture [8] compresses a source sequence into a single fixed-size encoder hidden state $h_T^{\text{enc}}$ , then conditions the decoder on that state. For long sentences the bottleneck is fatal: a 100-word English sentence has to fit into one $d$ -dim vector before the decoder reads any of it.

Bahdanau et al. [9] replaced the bottleneck with soft attention over the entire encoder hidden state sequence $h_1^{\text{enc}}, \dots, h_T^{\text{enc}}$ . At decoder step $t$ with decoder state $s_t$ , compute alignment scores

e_{t, j} = v^\top \tanh(W_a s_{t-1} + U_a h_j^{\text{enc}}),

normalize via softmax over $j$ ,

\alpha_{t, j} = \frac{\exp(e_{t, j})}{\sum_{j'} \exp(e_{t, j'})},

and form a context vector $c_t = \sum_j \alpha_{t, j} h_j^{\text{enc}}$ . Then condition the decoder step on $c_t$ in addition to $s_{t-1}$ and the previous output.

This is exactly the mechanism that survived (in scaled-dot-product form) into the Transformer [11]. The structural improvement is that the decoder can attend over the entire encoder output, so the gradient path from any decoder step to any encoder step is just one attention edge, not $T$ recurrent steps. The encoder bottleneck is gone, and effective context length jumps from “as much as can fit in $h_T^{\text{enc}}$ ” to “the entire input sequence.”

Implementations

Three idiomatic versions of the same LSTM cell: PyTorch’s one-liner nn.LSTM, a manual cell that exposes every gate, and a JAX/Flax sketch.

python

import torch
import torch.nn as nn

# One-liner: a stacked, bidirectional LSTM.
lstm = nn.LSTM(
  input_size=128,
  hidden_size=256,
  num_layers=2,
  bidirectional=True,
  batch_first=True,
)

# x: (batch, time, features). h0, c0 default to zeros.
x = torch.randn(32, 50, 128)
y, (hN, cN) = lstm(x)
# y: (32, 50, 512)   hN, cN: (4, 32, 256)  -- 4 = 2 layers * 2 directions
print(y.shape, hN.shape, cN.shape)

nn.LSTM is fused into a single cuDNN call when running on GPU. The 4-tuple of gate weights is packed into one parameter per layer.

Run a tiny manual LSTM in NumPy on a parity-counting toy task (output 1 if the running count of 1s in the input sequence is odd):

Idle (packages: numpy)

import numpy as np

def sigmoid(z):
  return 1.0 / (1.0 + np.exp(-z))

class TinyLSTM:
  def __init__(self, in_dim, h_dim, seed=0):
      rng = np.random.default_rng(seed)
      # Each W_* is shape (h_dim, in_dim + h_dim) for the [h_prev, x_t] concat.
      D = in_dim + h_dim
      self.W = rng.normal(0, 0.1, size=(4 * h_dim, D))
      self.b = np.zeros(4 * h_dim)
      # Forget-gate bias init = 1 to keep f_t near 1 at start.
      self.b[h_dim : 2 * h_dim] = 1.0
      # Output head.
      self.V = rng.normal(0, 0.1, size=(1, h_dim))
      self.h_dim = h_dim
      self.in_dim = in_dim

def step(self, x_t, h, c):
      z = self.W @ np.concatenate([h, x_t]) + self.b
      H = self.h_dim
      i_t = sigmoid(z[0:H]);    f_t = sigmoid(z[H:2*H])
      o_t = sigmoid(z[2*H:3*H]); g_t = np.tanh(z[3*H:4*H])
      c = f_t * c + i_t * g_t
      h = o_t * np.tanh(c)
      return h, c

def predict(self, xs):
      h = np.zeros(self.h_dim); c = np.zeros(self.h_dim)
      for x in xs:
          h, c = self.step(x, h, c)
      return sigmoid((self.V @ h)[0])

# Evaluate (untrained) on a few toy sequences just to confirm the pipeline runs.
np.random.seed(0)
m = TinyLSTM(in_dim=1, h_dim=8)
for _ in range(5):
  seq = np.random.binomial(1, 0.5, size=(10,)).astype(float).reshape(-1, 1)
  parity = int(seq.sum() % 2)
  p = m.predict(seq)
  print(f"sequence sum={int(seq.sum())} parity={parity}  predicted_p_odd={p:.3f}")

(no output yet)

Comparison

Architecture	Parameters per cell	Captures long deps	Parallel across time	Training stability	Dominant era
Vanilla RNN	~ d (d_in + d_h)	no (vanishing past ~10 steps)	no (sequential)	unstable	1990s, mostly historical
LSTM	~ 4 d (d_in + d_h)	yes (cell-state path)	no (sequential)	good with clipping	2014-2017 NLP and speech
GRU	~ 3 d (d_in + d_h)	yes (similar mechanism)	no (sequential)	good with clipping	2014-2017, smaller-budget niche
Transformer	~ 4 d^2 per layer (attn + FFN)	yes (any-to-any attention)	yes	good with warmup + LR schedule	2017-present
Bidirectional LSTM	2x LSTM cost	yes	no	good	2014-2018 sequence labelling
Stacked LSTM (2-3 layers)	L x LSTM cost	yes	no	requires layer/batch norm at depth	2014-2018 speech (DeepSpeech)

Real-world uses

Domain	Use	Why an RNN/LSTM/GRU works (or worked)
Machine translation (pre-2017)	seq2seq with attention (Bahdanau)	Variable-length input + variable-length output; attention removed the encoder bottleneck. Now Transformer dominant.
Speech recognition	Deep bidirectional LSTM with CTC loss (DeepSpeech, Listen Attend and Spell)	Streaming-friendly forward LSTM; bidirectional version when latency permits. Conformers (CNN + Transformer) now SOTA.
Time series forecasting	Demand, sensor, finance	Sequence with strong autocorrelation; LSTM is still a competitive baseline at small/medium scale.
Gesture & activity recognition	Sensor / video classification	Variable-length input from accelerometer or per-frame CNN features.
Online learning agents	Reinforcement learning policies with memory	LSTM hidden state acts as the agent's memory of partially observable history (DeepMind R2D2, IMPALA).

Pitfalls

Always clip the gradient

Without clipping, RNN training will eventually take a step of catastrophic magnitude on a sequence whose particular Jacobian product blows up. Standard recipe: clip the global gradient norm to 1 or 5 (torch.nn.utils.clip_grad_norm_(params, 1.0)). Adding clipping costs almost nothing and saves training runs [4].

Sequence length costs memory linearly

BPTT stores activations for every time step in order to compute gradients. A 1024-token sequence costs 1024x the activation memory of a single forward pass. Two production fixes: truncated BPTT (backprop only the last $K$ steps and detach the rest), and gradient checkpointing (recompute activations in segments at backward time, trading compute for memory). The same trade-off lives in Transformer training, but Transformers are at least parallel across time.

Teacher forcing vs autoregressive inference mismatch

Standard seq2seq training feeds the ground-truth previous token to the decoder (“teacher forcing”). At inference there is no ground truth, so the decoder is fed its own previous prediction. The distribution mismatch causes “exposure bias”: small errors compound. Mitigations include scheduled sampling (Bengio 2015) and minimum-risk training; in modern Transformer-based language models the issue is reduced because beam search and pretraining at scale dominate the sample-compounding effect.

Forget-gate bias init matters

If you initialize the LSTM forget-gate bias to zero, the gate sits at $\sigma(0) = 0.5$ and the cell state attenuates by $0.5^T$ over $T$ steps. The standard fix is to initialize $b_f$ to $1$ (or $2$ ), so $\sigma(b_f) > 0.7$ at start and gradients flow until the network learns selective forgetting. PyTorch’s nn.LSTM does not do this for you; check the framework defaults.

Pop quiz

Check yourself

Score: 0 / 5(0 answered)

1. Why does a vanilla RNN suffer from vanishing gradients?
2. Why does the LSTM cell-state recurrence c_t = f_t * c_{t-1} + i_t * c_tilde help with vanishing gradients?
3. When would you prefer a GRU over an LSTM?
4. What problem did Bahdanau attention solve in seq2seq?
5. What does gradient clipping do, and why is it standard for RNN training?

Interview Q&A

Derive the LSTM cell-state equation from the design objective of preserving long-range gradients.research

Objective. Build a recurrence $c_t = g(c_{t-1}, x_t)$ such that $\partial c_t / \partial c_{t-1}$ is close to the identity along most paths, even after many time steps.

Step 1: additive update. The simplest such recurrence is $c_t = c_{t-1} + \Delta_t$ , which gives $\partial c_t / \partial c_{t-1} = I$ exactly. But pure addition has no mechanism to forget or to control how much new information enters.

Step 2: gate the previous state. Replace the constant $1$ multiplier on $c_{t-1}$ with a learned gate $f_t \in (0, 1)^d$ :

c_t = f_t \odot c_{t-1} + \Delta_t.

When $f_t \to 1$ on the dimensions that matter, the cell-state gradient is $\mathrm{diag}(f_t) \approx I$ and gradients flow. When $f_t \to 0$ on a dimension, that dimension is forgotten.

Step 3: gate the new information. $\Delta_t$ should not blow up the cell state, so multiply it by an input gate $i_t \in (0, 1)^d$ and scale the actual update through a $\tanh$ :

\Delta_t = i_t \odot \tilde c_t, \qquad \tilde c_t = \tanh(W_c [h_{t-1}, x_t] + b_c).

Step 4: gate the output. The cell state is the long-term memory; the network should be able to expose only part of it at each step. Add an output gate $o_t \in (0, 1)^d$ :

h_t = o_t \odot \tanh(c_t).

Step 5: each gate is a small sigmoid layer of $[h_{t-1}, x_t]$ .

i_t = \sigma(W_i [h_{t-1}, x_t] + b_i), \quad f_t = \sigma(W_f [h_{t-1}, x_t] + b_f), \quad o_t = \sigma(W_o [h_{t-1}, x_t] + b_o).

Result. The LSTM [5], with the forget gate added by Gers, Schmidhuber, Cummins [6]. Initialize $b_f$ to $1$ so $f_t \approx 0.7$ at start and the cell-state path is approximately identity from training step zero.

Why does the cell state, and not the hidden state, carry long-range information in an LSTM?onsite

The hidden state is squashed by $\tanh$ at every time step ( $h_t = o_t \odot \tanh(c_t)$ ). The $\tanh$ is bounded and has gradient at most 1, so any path through $h_t \to h_{t+1}$ saturates exactly like a vanilla RNN’s $\tanh$ recurrence: gradients shrink geometrically through it.

The cell state is additive and gated: $c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t$ , with no $\tanh$ applied to $c_{t-1}$ along the recurrence. The gradient through $c$ at one step is $\mathrm{diag}(f_t)$ plus small terms, and the network can learn to keep $f_t$ near 1 on the dimensions that should remember, so gradients flow through the cell state with negligible attenuation.

The architectural decoupling is the point: $c$ is the unbounded long-term memory and propagates gradients; $h$ is the squashed short-term view exposed at each step. Vanilla RNNs collapse the two and pay for it with vanishing gradients.

When would you prefer a GRU over an LSTM?phone

Three reasons that all reduce to “smaller and cheaper”:

Parameter budget. A GRU has $\sim 3$ gate matrices vs an LSTM’s $\sim 4$ , so it is about 25% smaller per cell at the same hidden size. On edge devices, mobile, or any place where memory or quantization matter, GRU is the cheaper choice.
Smaller datasets. Empirically GRU and LSTM are within noise on most tasks [7], but the smaller GRU can mildly resist overfitting on small datasets.
Faster wall-clock. Fewer matrix multiplies per step. On GPU the difference is usually less than 25% (memory bandwidth dominates) but on CPU it can be 1.3-1.5x.

LSTM is the safer default at large scale and on stateful tasks (where the explicit cell state has been observed to help). For most modern work, both are largely displaced by Transformers; GRU/LSTM linger in streaming contexts and small-budget edge models.

Why did attention 'fix' the seq2seq encoder bottleneck?onsite

The original seq2seq [8] compressed the entire source sentence into one fixed-size vector $h_T^{\text{enc}}$ . Two structural problems:

Capacity bottleneck. A 100-word sentence has to fit into one $d$ -dim vector. Empirically translation BLEU degraded sharply on sentences past about 30-40 words.
Gradient bottleneck. The gradient from any decoder step to any encoder step has to pass through the chain $h_T^{\text{enc}} \leftarrow h_{T-1}^{\text{enc}} \leftarrow \cdots$ , paying the vanishing-gradient tax.

Bahdanau attention [9] replaces the bottleneck with a learned weighted average over the entire encoder hidden state sequence: at each decoder step $t$ , compute alignment scores $e_{t, j}$ , normalize via softmax to get $\alpha_{t, j}$ , and form context $c_t = \sum_j \alpha_{t, j} h_j^{\text{enc}}$ . The decoder reads from anywhere in the encoder at any time.

Two consequences:

No more capacity bottleneck. The decoder sees the full encoder sequence at each step.
One-step gradient path. From decoder step $t$ to encoder step $j$ , the gradient flows through one attention edge. No T-deep chain to traverse.

This is exactly the mechanism the Transformer [11] generalized (multi-head, scaled dot-product, applied to self-attention in addition to cross-attention). Once you accept the attention principle, the only remaining role for the recurrence is sequential bookkeeping, which is itself the parallelism bottleneck Transformers eliminated.

Build it yourself

A weekend project to internalize the recurrence:

Implement the manual LSTM cell from the CodeCompare above in PyTorch (no nn.LSTM). Train on parity counting over sequences of length 50 and reach 99% test accuracy.
Replace the LSTM cell with a vanilla RNN cell. Watch it fail at length 50 (it will plateau around 50% accuracy). Quantify how the failure threshold scales with sequence length.
Add gradient clipping and confirm it stabilizes the vanilla RNN on shorter sequences.
Stretch: swap the LSTM cell for a GRU cell, confirm accuracy is similar with ~25% fewer parameters, and measure the wall-clock speedup.
Stretch further: add Bahdanau attention on top of an encoder-decoder LSTM and reproduce the 30-40 word BLEU improvement on a small WMT translation subset.

References

Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature 323, 533-536. doi:10.1038/323533a0
Werbos, P. J. (1990). Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10), 1550-1560. doi:10.1109/5.58337
Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 157-166. doi:10.1109/72.279181
Pascanu, R., Mikolov, T., and Bengio, Y. (2013). On the difficulty of training recurrent neural networks. ICML 2013. arXiv:1211.5063
Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation 9(8), 1735-1780. doi:10.1162/neco.1997.9.8.1735
Gers, F. A., Schmidhuber, J., and Cummins, F. (2000). Learning to forget: Continual prediction with LSTM. Neural Computation 12(10), 2451-2471. doi:10.1162/089976600300015015
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. EMNLP 2014. arXiv:1406.1078
Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS 2014. arXiv:1409.3215
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015 (arXiv preprint 2014). arXiv:1409.0473
Graves, A., Mohamed, A.-R., and Hinton, G. (2013). Speech Recognition with Deep Recurrent Neural Networks. ICASSP 2013. arXiv:1303.5778
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org

Interactive Demo

Compact View

h_t = tanh(Wx + Uh_t-1)

Unrolled View

RNN

the

h_t

h_1

RNN

cat

h_t

h_2

RNN

sat

h_t

h_3

RNN

h_t

h_4

RNN

mat

h_t

Press Step to process the first token

Tip: Watch the colored hidden state change as each word is processed - this is the RNN memory

Convolutional Neural Networks Autoencoders & VAEs