03 / Resources/Deep Learning · Recurrent Networks

Interactive Diagnostic Essay · Topic 09

Recurrent Networks: Memory in Motion

CNNs see everything at once — sentences, audio and stock prices unfold. Watch a network read word by word while carrying a running summary of everything it has read, and meet the LSTM that learns what to remember and what to forget.

01 / One Cell, Many Timesteps

The Network That Reads in a Loop

A dense layer would need fixed-length input and zero memory between words. An RNN instead processes one token at a time, keeping a hidden state $h_t$ — a compressed summary of the story so far:

h_t = \tanh(W_{xh}\, x_t + W_{hh}\, h_{t-1} + b)

y_t = W_{hy}\, h_t

The secret is weight sharing across time: the same $W_{xh}, W_{hh}, W_{hy}$ process every word. Unrolled, the single cell becomes a chain — watch the sentence “The cat sat down” flow through it in the animation, one glowing timestep at a time.

After the last word, $h_4$ holds the sentence's meaning — ready for sentiment, translation, or the next-word prediction behind every language model.

UNROLLED RNN · SENTENCE = “THE CAT SAT DOWN”looping

The

↓

cat

↓

sat

↓

down

↓

h1 ⟶

h2 ⟶

h3 ⟶

same weights at every step · the arrow between cells is the memory

02 / Training Through Time

Backpropagation Through Time — and the Vanishing Gradient

ERROR SIGNAL FLOWING BACKWARDfading as it travels

by h₁ the signal is nearly gone — early words barely learn

To train the shared weights, we unroll the loop into the chain above and backpropagate through every timestep at once — BPTT. But each backward hop multiplies the gradient by $W_{hh}$ and the tanh derivative (≤ 1):

The Product That Breaks RNNs

\frac{\partial L}{\partial h_1} = \frac{\partial L}{\partial h_T} \prod_{t=2}^{T} W_{hh}^T \, \text{diag}(\tanh' (h_t))

Vanishing: factors < 1 compound — the signal decays to nothing and early timesteps never learn long-range patterns.
Exploding: factors > 1 compound — the signal blows up to NaN. The cheap fix is gradient clipping: cap the norm before the update.

Vanishing can't be fixed by clipping — it needs an architecture that lets information bypass the multiplications. Enter the LSTM.

03 / Gated Memory

LSTM: Learn to Remember, Learn to Forget

The Long Short-Term Memory cell adds a cell state — a conveyor belt running along the whole sequence — guarded by three learned gates (each a tiny sigmoid layer outputting values in $[0, 1]$ , i.e. “how much to let through”):

1 · FORGET GATE

f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)

How much of the old memory to keep — “the subject changed, drop the old gender.”

2 · INPUT GATE

i_t = \sigma(W_i [h_{t-1}, x_t] + b_i)

How much of the new candidate

\tilde{C}_t

to write into memory.

3 · OUTPUT GATE

o_t = \sigma(W_o [h_{t-1}, x_t] + b_o)

How much memory to reveal as this step's

h_t

CELL STATE UPDATE:

C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t

The addition in $C_t$ is the magic: the gradient can flow additively along the cell state for hundreds of steps without being repeatedly multiplied — the vanishing problem simply doesn't apply to the belt.

LSTM CELL · GATES FIRE IN SEQUENCElooping

forget → input → output · three tiny sigmoid decisions per timestep

04 / The Family Tree

GRU, Bi-RNNs & Where Sequences Live Today

Vanilla RNN

One matrix of weights, no gates. Cheap, fast — but forgets within ~10 steps. Fine for tiny sequences.

GRU

LSTM's leaner sibling: merges forget+input into one update gate. Nearly as good, 25% fewer parameters.

Bi-RNN

Two RNNs read the sequence in both directions and concatenate — 'bark' means different things after 'dog' or after 'tree'.

Seq2Seq

Encoder RNN compresses the input into a vector; decoder RNN generates the output — the blueprint of machine translation.

WHERE RNNS STILL WORK

· Time-series forecasting (energy, demand, sensors)
· Streaming / online inference on devices
· Small-data regimes where transformers overfit
· Speech frontend features

THE SEQUENTIAL BOTTLENECK

RNNs must process token $t$ before token $t+1$ — no parallelism across time, and long-range memory still fades. The Transformer (2017) threw away recurrence for self-attention: everything in parallel, every token directly connected to every other. That is the architecture behind GPT — but its attention scores are exactly the weighted sums, sigmoids and softmaxes you have met across Topics 02–09.

05 / Training Mechanics

Backprop Through Time: One Weight, Many Gradients

Section 02 named the algorithm; here is the mechanism. BPTT is nothing more exotic than unrolling the recurrence into a chain — $h_1, h_2, \dots, h_T$ — treating each timestep as an ordinary node, and running the plain backpropagation from Topic 07 on that unrolled graph.

The crucial part is that the copies are not separate parameters. Every cell in the chain is the same $W_{hh}$ , so its gradient is the sum of the contributions from every timestep:

The Shared Weight Accumulates

\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial L_t}{\partial W_{hh}}, \quad L = \sum_{t=1}^{T} L_t

Each term is itself a product of local Jacobians travelling back from $t$ to the end. Multiply long enough and the magnitude of that product decides everything: slightly below 1 and it collapses toward zero (vanishing), slightly above 1 and it blows past the representable range (exploding).

Exploding → gradient clipping: rescale the gradient whenever its norm exceeds a threshold, before the update.
Vanishing → gating: an additive cell-state path (LSTM, Topic 03) that carries the signal across many steps without repeated multiplication.

GRADIENT MAGNITUDE AT EACH TIMESTEP9s loop · vanilla → gated

vanilla bars shrink toward t=1 · gated bars stay level · signal sweeps right to left

← Topic 08: Convolutional Networks Topic 10: Random Forests →