RNNs, LSTMs and Attention Explained With Real Numbers

Dog bites man. Man bites dog. Exactly the same three words, and completely different meanings. Order carries information, and the networks we have built so far, which treat inputs as fixed-size bundles, have no natural way to respect it. Today we fix that.

• Dog bites man
• Man bites dog
• Same words, new meaning

In the last episode we used PyTorch to build convolutional networks that read images. In this episode, we build networks that read sequences: text, audio, sensor streams, anything where one step depends on the steps before it. We will cover recurrent networks, LSTMs and GRUs, and finally attention.

• Last time: PyTorch and CNNs
• Today: sequences
• Memory is the new idea

Here is the plan. First, what makes data sequential. Then the recurrent neural network and its hidden state, worked out with real numbers. Then backpropagation through time and the vanishing gradient problem, which explains why LSTMs exist. We finish with practical training tricks, and the bottleneck that motivates attention.

• RNN and hidden state
• Vanishing gradients
• LSTM, GRU, attention

Sequential data is any data where order matters and length varies. A sentence has maybe five words or five hundred. A stock series has one value per day. A strand of DNA has thousands of letters. A standard network wants a fixed number of inputs, so it already struggles here.

• Text, audio, prices, DNA
• Order matters
• Length varies

Suppose we flatten a sentence into one big vector and feed it to a dense network. Two problems appear. The input size is fixed, so longer sentences do not fit. And the weights that detect the word not at position three are different from the weights at position seven, so the network must relearn the same idea at every position.

• Fixed input size
• No shared weights over time
• Position-specific learning

The fix has two ingredients. First, reuse the same small set of weights at every time step, just like a convolution reuses a kernel across an image. Second, carry a running summary of everything seen so far. That summary is called the hidden state, and it is the heart of every model in this video.

• Same weights at every step
• Carry a summary forward
• That summary is the hidden state

Here is the whole recurrent network in one line. The new hidden state h t equals tanh of W x times the current input x t, plus W h times the previous hidden state h t minus one, plus a bias b. The tanh squashes the result between minus one and one. At the start, h zero is just a vector of zeros.

• h_t = tanh(Wx x_t + Wh h_{t-1} + b)
• Hidden state updates each step
• h_0 starts at zero

A loop is hard to picture, so we unroll it. Draw one copy of the cell for each time step, and connect the hidden state from each copy to the next. The weights are identical in every copy. Notice what this means: an RNN reading a hundred words is effectively a network a hundred layers deep.

• Loop becomes a chain
• Same weights in every copy
• Depth equals sequence length

Let us use a scalar hidden state so we can compute by hand. Set w x to zero point five, w h to zero point eight, and the bias to zero. The inputs are one, two, then minus one. At step one, h zero is zero, so the pre-activation is zero point five times one, which is zero point five. The tanh of zero point five is about zero point four six.

• w_x=0.5, w_h=0.8, b=0
• x1=1, h0=0
• h1=tanh(0.5)≈0.46

Step two. The input is two, so the input part is one point zero. The memory part is zero point eight times zero point four six, about zero point three seven. Add them for one point three seven, and tanh gives about zero point eight eight. Step three. The input is minus one, giving minus zero point five, plus zero point eight times zero point eight eight, about zero point seven. The sum is zero point two, and h three is about zero point two.

• Step 2: 1.0 + 0.8×0.46 = 1.37
• h2 = tanh(1.37) ≈ 0.88
• Step 3: h3 ≈ 0.20

Look at what happened. The final value of zero point two was influenced by all three inputs, because each hidden state flowed into the next. But notice it is a lossy summary. We cannot recover the original sequence from one number, and with a real vector of, say, two hundred fifty six numbers, the same squeeze applies to every sentence, short or long.

• h3 depends on x1, x2, x3
• Compressed, lossy summary
• Fixed size, always

The same cell supports different task shapes. Many to one reads a whole sequence and outputs a single answer, like sentiment. Many to many produces an output at every step, like tagging each word. One to many starts from a single input and generates a sequence, like captioning an image. The difference is only where we attach the loss.

• Many to one
• Many to many
• One to many

How do we train this? Exactly as in episode twenty six. Unroll the network, run the forward pass, compute the loss, and apply the chain rule backwards. Because the same weights appear at every step, the gradient for W h is the sum of its contributions from every time step. This procedure is called backpropagation through time.

• Unroll, then backprop
• Sum gradients over all steps
• Called BPTT

Here is the key piece. To send a gradient from step T back to an earlier step k, we multiply many local derivatives together, one for each step in between. Each factor is roughly W h times one minus h squared, the derivative of tanh. So the signal reaching the past is a long product, and long products behave dramatically.

• dL/dh_k = dL/dh_T × ∏ dh_j/dh_{j-1}
• Each factor: W_h × (1 − h²)
• A long product of terms

Unrolling over a document with ten thousand tokens would eat memory. The standard shortcut is truncated backpropagation through time. We split the sequence into windows of, say, one hundred steps, carry the hidden state forward between windows, but stop the gradient at each window boundary. The model still sees context, but learning only looks back one window.

• Full unroll is costly
• Cut into windows
• Carry state, stop gradient

A classic bug in PyTorch. If you carry the hidden state between batches without detaching it, the computation graph keeps growing, memory climbs steadily, and training eventually crashes. Call detach on the hidden state at the start of each window. It keeps the value but cuts the graph, which is exactly what truncation means.

• Hidden state keeps its graph
• Memory grows every batch
• Detach between windows

Suppose each factor in that product is about zero point five. After ten steps, the gradient is zero point five to the tenth power, roughly zero point zero zero one. After twenty steps it is about one in a million. The loss at the end of a sentence barely whispers to the words at the beginning, so the network cannot learn long-range dependencies.

• Factor 0.5 per step
• 0.5 to the 10 ≈ 0.001
• Early steps get no signal

Two things shrink the product. The derivative of tanh is at most one, and it drops toward zero whenever the unit saturates near plus or minus one. And if the recurrent weight is below one in size, it shrinks the signal further. Both effects multiply together, so vanishing is the default behavior, not a rare accident.

• tanh derivative ≤ 1
• Saturation gives near zero
• Weights also matter

The product can also grow. If each factor is one point five, then after ten steps the gradient is about fifty eight times larger, and after twenty steps more than three thousand times larger. Updates become enormous and the loss turns into not a number. This is the exploding gradient problem.

• Factor 1.5 per step
• 1.5 to the 10 ≈ 58
• 1.5 to the 20 ≈ 3,300

Exploding gradients have a simple, effective cure: gradient clipping. Compute the norm of the whole gradient. If it exceeds a threshold, say five, scale the gradient down so its norm equals five. The direction stays the same, only the step length is limited. Vanishing gradients, however, have no such quick fix, and that motivates a new architecture.

• Cap the gradient norm
• If norm > 5, rescale
• Direction preserved

What do we really want? A path along which information and gradients can travel many steps without being squashed or multiplied by small numbers each time. Think of a conveyor belt running through the whole sequence, where we can add or remove items deliberately, rather than rewriting the entire contents at every step. That is the idea behind the LSTM.

• Need a path without squashing
• Multiply by near 1
• Additive memory

The long short-term memory network keeps two vectors. The cell state c t is the conveyor belt, the long-term memory. The hidden state h t is what the cell shows to the outside world at each step. Three gates, each a sigmoid that outputs numbers between zero and one, control what happens: a forget gate, an input gate, and an output gate.

• Cell state c_t: long-term memory
• Hidden state h_t: output
• Three gates

A gate is just a sigmoid output multiplied element by element with some signal. If the gate is zero, the signal is blocked. If it is one, the signal passes untouched. Values in between let through a fraction. Because the network computes the gate values from the current input and previous hidden state, it learns when to open and close them.

• Sigmoid outputs 0 to 1
• 0 blocks, 1 passes
• Multiply element-wise

The central equation is this. The new cell state c t equals the forget gate f times the old cell state, plus the input gate i times a candidate value g, where g is a tanh of the current input and previous hidden state. Forget a fraction of the old, add a fraction of the new. Crucially, this is an addition, and that is what saves the gradient.

• c_t = f ⊙ c_{t-1} + i ⊙ g
• Forget old, write new
• Addition, not rewriting

Let us compute one step. The old cell state is two. The forget gate is zero point nine, the input gate zero point five, and the candidate is zero point six. The new cell state is zero point nine times two, which is one point eight, plus zero point five times zero point six, which is zero point three. That gives two point one. With an output gate of zero point seven, the hidden state is zero point seven times tanh of two point one, about zero point six eight.

• c_prev=2.0, f=0.9, i=0.5, g=0.6
• c = 1.8 + 0.3 = 2.1
• h = 0.7 × tanh(2.1) ≈ 0.68

Now change one number. If the forget gate drops to zero point one, the old memory contributes only zero point two, and the new cell state is zero point five. The network just erased most of its past. This is useful: when a sentence ends or a new topic begins, the model can reset, and when a detail matters, it can hold on with a forget gate near one.

• f = 0.1 erases memory
• c = 0.2 + 0.3 = 0.5
• Context-dependent reset

Look at the path along the cell state. The derivative of c t with respect to c t minus one is mainly just the forget gate. If the network learns forget gates near one, the product over ten steps is not zero point five to the tenth, about zero point zero zero one, but zero point nine to the tenth, about zero point three five. The gradient survives, and the model can learn long-range patterns.

• dc_t/dc_{t-1} ≈ f
• Product of forget gates
• 0.9 to the 10 ≈ 0.35

The gated recurrent unit is a leaner cousin of the LSTM. It merges the cell state and hidden state into one vector and uses two gates: an update gate z and a reset gate r. With fewer parameters, it trains a little faster, and in practice it often performs similarly. Which one wins depends on the task.

• Two gates: update and reset
• No separate cell state
• Fewer parameters

The GRU update is a blend. The new state equals one minus z times the old state, plus z times the candidate. Take z of zero point three, an old state of zero point eight, and a candidate of minus zero point two. The old part gives zero point seven times zero point eight, which is zero point five six. The new part gives zero point three times minus zero point two, which is minus zero point zero six. The result is zero point five.

• h = (1−z)·h_prev + z·h_cand
• z=0.3, h_prev=0.8, cand=−0.2
• h = 0.56 − 0.06 = 0.50

Time to put these cells to work. For sequence classification, such as deciding if a movie review is positive, we feed in the words one at a time and take the final hidden state, pass it through a linear layer and a sigmoid or softmax, and apply cross entropy loss. Averaging or taking the maximum of all hidden states often works even better than using only the last one.

• Read the whole sequence
• Classify from final state
• Or pool over all states

A language model predicts the next token given all the previous ones. At each step, the hidden state goes through a linear layer into a softmax over the entire vocabulary. The target is simply the next word in the text, so no human labels are needed. The text labels itself, which is why this approach scales so well.

• Predict the next token
• Softmax over vocabulary
• Loss: cross-entropy

We measure language models with perplexity, which is the exponential of the average cross-entropy loss. A model guessing uniformly over ten thousand words has a loss of the natural log of ten thousand, about nine point two one, so its perplexity is ten thousand. If a trained model reaches a loss of three, its perplexity is about twenty, as if it were choosing among twenty equally likely words.

• Perplexity = exp(loss)
• Random guess on 10,000 words: loss 9.21
• Loss 3.0 means perplexity ≈ 20

During training, what do we feed as input at step t? With teacher forcing, we feed the true previous word from the data, not the model's own guess. If the model predicted the wrong word at step two, step three still sees the correct one. Training becomes much more stable, because errors do not snowball.

• Feed the true previous word
• Fast, stable training
• Parallel loss over steps

But at test time there is no teacher. The model must feed its own previous prediction back in. If it makes an early mistake, it now sees an input it never saw in training, and errors can compound. This train-test mismatch is called exposure bias. Remedies include scheduled sampling, where we occasionally feed the model's own guesses during training.

• At test time, no teacher
• Own mistakes feed forward
• Train and test mismatch

Real data brings a practical issue. A batch is a rectangular tensor, but sentences have different lengths. Say three sentences have lengths three, five and two. We pad the shorter ones with a special padding token up to length five, so the batch becomes a three by five grid. But now the padding must not influence learning.

• Batches need equal length
• Pad short ones with zeros
• Lengths 3, 5, 2 become 5

The solution is a mask, a grid of ones for real tokens and zeros for padding. Multiply the per-token loss by the mask, sum it, and divide by the number of real tokens. Here we have ten real positions out of fifteen. Divide by ten, not fifteen. Otherwise, padded batches silently shrink the loss and distort the gradients, a very common bug.

• Mask: 1 real, 0 padding
• 10 real tokens of 15
• Divide by 10, not 15

Masking the loss is not enough for classification. If we take the final hidden state of a padded row, we would be reading the state after many padding steps. Use packed sequences, or pick the hidden state at each sentence's true last position. When the whole input is available, a bidirectional RNN also reads the sequence backwards and concatenates both states, giving each word context from both sides.

• Packed sequences skip padding
• Final state at true last token
• Bidirectional reads both ways

Translation needs a different shape: read a whole sentence, then write another one of a different length. The sequence to sequence model does this with two networks. An encoder RNN reads the source sentence and compresses it into its final hidden state. A decoder RNN starts from that state and generates the target sentence one word at a time.

• Encoder reads the source
• Decoder writes the target
• Translation example

Here is the flaw. Whether the source sentence has five words or fifty, the encoder must pack everything into one fixed-size vector. That vector has to carry the meaning, the names, the numbers, and the grammar. Early words get overwritten, and translation quality tends to fall as sentences get longer. One vector is a bottleneck.

• Everything squeezed into one vector
• Same size for 5 or 50 words
• Quality drops on long inputs

The insight is simple. The encoder already produced a hidden state for every source word. Why throw all but the last away? Keep them all, and let the decoder look back at them whenever it needs. When writing a word, it can focus on the relevant source words, the way a translator glances back at the original sentence.

• Keep all encoder states
• Decoder chooses where to look
• Like consulting notes

Here is the mechanism with numbers. The decoder forms a query vector, say one comma zero. Each encoder state provides a key: two zero, zero one, and one one. We score each key by its dot product with the query, a measure of alignment from episode three. The scores are two, zero, and one.

• Query q=[1,0]
• Keys [2,0], [0,1], [1,1]
• Scores 2, 0, 1

Raw scores are not weights, so we apply softmax. The exponentials are about seven point three nine, one, and two point seven two, summing to eleven point one one. Dividing gives attention weights of roughly zero point six seven, zero point zero nine, and zero point two four. They are positive and sum to one, so they are a distribution over source words.

• e^2=7.39, e^0=1, e^1=2.72
• Sum = 11.11
• Weights 0.67, 0.09, 0.24

Each source position also carries a value. Suppose they are ten, twenty, and thirty. The context is the weighted average: zero point six six five times ten, plus zero point zero nine times twenty, plus zero point two four five times thirty, which comes to about fifteen point eight. The decoder receives this fresh, step-specific summary, which changes at every output word.

• Values 10, 20, 30
• 0.67×10 + 0.09×20 + 0.24×30
• Context ≈ 15.8

Plot the weights for every output word and you get an alignment map, which in translation often lights up along a diagonal, matching words across languages. Attention also gives gradients a direct shortcut to any source word, no matter how far back. In practice, dot products are divided by the square root of the vector dimension, so that softmax does not saturate.

• Weights form an alignment map
• Gradient shortcut to any word
• Real models scale the dot product

Let us recap. A recurrent network reuses one cell across time and carries a hidden state. Training unrolls it, which is backpropagation through time, where long products of factors make gradients vanish or explode. Clipping tames explosions, and LSTM and GRU gates give gradients a safer path. We also covered teacher forcing, padding and masking, and attention as the remedy for the one-vector bottleneck.

• RNN: shared weights and hidden state
• BPTT and vanishing gradients
• LSTM, GRU, masking, attention

Here is a thought to carry forward. If attention can reach any word directly, do we still need the recurrent chain at all? The answer led to the Transformer, which keeps attention and drops recurrence, and it powers modern language models. That is the next episode.

• Attention without recurrence
• Parallel over all tokens
• Episode 29