Transformers Explained: Attention, Math and How LLMs Really Work

Every modern chatbot, translator and code assistant you have used is built on one architecture: the transformer. It has no loops and no memory cells. It just lets every word look at every other word and decide what matters. In this video we build it piece by piece, with real numbers.

• Chatbots, translators, code assistants
• All built on transformers

Last time we saw recurrent networks read text one step at a time, squeezing the past into a single hidden state, and we saw attention rescue them by letting the decoder look back. The transformer asks a bold question: what if attention is all we need, and we throw the recurrence away?

• Episode 28: RNNs, LSTMs, attention
• Problem: sequential bottleneck

Our roadmap has four stops. First, how text becomes numbers. Second, the attention formula, derived with a tiny worked example. Third, how the full block and training work. Finally, fine-tuning, LoRA, inference cost, and what these models still get wrong.

• Tokens to attention to training
• Then LoRA, cost, limits

A neural network needs numbers, so we must chop text into pieces. Characters make sequences very long. Whole words make the vocabulary enormous and fail on new words like typos or names. The compromise is subwords: common words stay whole, rare words split into reusable fragments.

• Letters: sequences too long
• Words: vocabulary too huge

Byte pair encoding builds the vocabulary automatically. Start with single characters. Count which adjacent pair occurs most often, merge it into a new symbol, and repeat. If 'l' and 'o' appear together most, they become 'lo'. Then 'lo' and 'w' may become 'low'. After thousands of merges you get a vocabulary of common chunks.

• Start with characters
• Merge most frequent pair
• Repeat

Each token then maps to an integer ID, an index in the vocabulary, often tens of thousands of entries or more. This explains odd behavior. A model sees a word as a few chunks, not as letters, so counting letters or doing digit arithmetic can be surprisingly hard for it.

• Each token gets an integer ID
• Spelling and arithmetic suffer

An integer has no geometry, so we replace each ID with a learned vector. Imagine an embedding matrix with V rows, one per token, and d columns. Looking up token 389 simply grabs row 389. If d is 4096, each token becomes a point in a four thousand dimensional space, and training moves these points around.

• Lookup table of size V by d
• Learned during training

After training, similar tokens end up near each other, and directions can encode relations. The classic example is that the vector for king minus man plus woman lands near queen. That is a famous illustration from word embeddings, and it shows geometry doing the work of meaning, though real behavior is messier.

• Similar words cluster
• Directions carry relations

Here is a catch. Attention treats its inputs as a set. If we shuffle the words, the raw computation is unchanged. But dog bites man and man bites dog mean very different things. We must inject word order into the vectors ourselves.

• Dog bites man vs man bites dog
• Same bag of vectors

The original transformer adds a position vector to each embedding. Each dimension is a sine or cosine wave of the position, at a different frequency. Fast waves distinguish neighbors, slow waves distinguish far apart positions, like the hands of a clock. Together, every position gets a unique fingerprint.

• Sine and cosine of different frequencies
• Add to the embedding

Many models instead learn a position vector per slot, and many modern ones use rotary embeddings, which rotate the query and key vectors by an angle that depends on position, so attention naturally depends on relative distance. The details vary, but the goal is constant: make order visible to attention.

• Learned positions
• Rotary embeddings
• Relative distance

Now the heart of the model. Think of a library search. Each token produces a query, describing what it is looking for. Each token also produces a key, advertising what it contains, and a value, the actual content it will hand over. A query that matches a key pulls in that token's value.

• Query: what I seek
• Key: what I offer
• Value: what I give

In math, stack the n input vectors into a matrix X with n rows and d columns. Multiply by three learned weight matrices to get Q equals X times W Q, K equals X times W K, and V equals X times W V. Same input, three different roles, and the network learns what each role should contain.

• Q = XW_Q, K = XW_K, V = XW_V
• Same input, three roles

How do we measure whether a query matches a key? With the dot product, which from episode three we know measures alignment. Vectors pointing the same way give a large positive number, perpendicular vectors give zero. Doing this for every query against every key gives a score matrix: Q times K transpose, of shape n by n.

• Score = q dot k
• Aligned vectors score high

Raw scores can be any size, so we pass each row through softmax, turning scores into positive weights that sum to one. The output for a token is then the weighted average of the value vectors. Big score, big weight, more of that token's content flows in. Put together: attention of Q, K, V equals softmax of Q K transpose, times V.

• Scores to probabilities
• Each row sums to one

There is a subtle flaw. Suppose query and key components are independent, each with mean zero and variance one. A dot product adds d k products. Each product has variance one, and variances of independent terms add, so the dot product has variance d k and standard deviation square root of d k.

• Dot product sums d_k terms
• Variance grows with d_k

With d k equal to sixty four, typical scores are around plus or minus eight. Softmax of numbers that spread out becomes nearly one-hot: one weight close to one, the rest near zero. In that saturated region the gradient is almost zero, so learning stalls. The fix must keep scores at a sane size.

• Softmax saturates
• Gradients vanish

Divide each score by square root of d k. Scaling a variable by c scales its variance by c squared, so the variance becomes d k divided by d k, which is one. No matter how large the key dimension, scores keep unit variance. That gives the scaled dot-product attention formula: softmax of Q K transpose over root d k, times V.

• Var(s/√d_k) = d_k / d_k = 1
• Scores return to unit scale

Let us compute one by hand with d k equal to four. The query is one, one, one, one. Three keys: key one is two, one, one, zero. Key two is one, one, zero, zero. Key three is one, minus one, zero, zero. The values are one zero, zero one, and one one.

• d_k = 4, three tokens
• q = [1,1,1,1]

Dot products first. Query with key one: two plus one plus one plus zero is four. With key two: two. With key three: one minus one is zero. Square root of four is two, so the scaled scores are two, one, and zero.

• q·k1 = 4, q·k2 = 2, q·k3 = 0
• Divide by √4 = 2 → 2, 1, 0

Exponentiate: e squared is 7.389, e to the one is 2.718, e to the zero is one. The sum is 11.107. Dividing gives weights 0.665, 0.245 and 0.090. The output is 0.665 times one zero, plus 0.245 times zero one, plus 0.090 times one one, which gives 0.755 and 0.335.

• e², e¹, e⁰ = 7.389, 2.718, 1
• Weights 0.665, 0.245, 0.090

Skip the scaling and scores of four, two, zero give weights 0.867, 0.117 and 0.016. Already sharper, and that is with a tiny d k of four. At realistic sizes the distribution collapses to one-hot and gradients vanish. A common mistake is forgetting that the scale factor depends on key dimension, not on sequence length.

• Scores 4, 2, 0 → 0.867, 0.117, 0.016
• Sharper, and worse at scale

Computed for a whole sentence, the weights form a heatmap. Row i shows how token i distributes its attention across all tokens. In 'the animal did not cross the street because it was tired', the row for 'it' can light up on 'animal'. Heatmaps are suggestive, but they are not a full explanation of the model's reasoning.

• Row i: where token i looks
• Rows sum to 1

One attention pattern is limiting, since a word may need to track grammar, reference and topic at once. Multi-head attention runs h attention operations in parallel, each with its own W Q, W K and W V on a smaller slice of dimension d divided by h. For example, d equal to 512 and eight heads gives each head sixty-four dimensions.

• h heads, each d_k = d/h
• Different relations in parallel

The head outputs are concatenated side by side, restoring width d, then multiplied by an output matrix W O that mixes information across heads. Total compute stays similar to one big head, but the model gets several specialized views at the same cost.

• Concat heads back to d
• Multiply by W_O

Deep stacks are hard to train, so each sublayer is wrapped in a residual connection: output equals x plus sublayer of x. The input skips around the layer on a shortcut. In backpropagation the derivative of x plus f of x contains a plain one, so gradients flow straight down even through dozens of layers.

• x + Sublayer(x)
• Gradients flow along the shortcut

Layer norm keeps activations well behaved. Take one token's vector two, four, six, eight. The mean is five. The variance is nine plus one plus one plus nine, over four, which is five, so the standard deviation is about 2.236. Subtract the mean and divide: minus 1.34, minus 0.45, 0.45, 1.34. Learned scale and shift parameters then restore flexibility.

• Take x = [2, 4, 6, 8]
• Mean 5, std √5 ≈ 2.236

Attention moves information between tokens. The feedforward network then processes each token independently: expand from d to roughly four d, apply a nonlinearity such as GELU, and project back to d. Much of a transformer's parameters live here, and many researchers view these layers as where factual knowledge is stored, though that is still an active research area.

• Two linear layers with nonlinearity
• Hidden size about 4d

One block is attention plus feedforward, each with a residual and a layer norm. Stack that block dozens of times and you have a transformer. Many modern models apply the norm before each sublayer, called pre-norm, which tends to train more stably than the original arrangement.

• Attention + FFN, repeated N times
• Pre-norm is common today

When generating text, a token must not see the future, or training would be cheating. We apply a causal mask: before softmax, set the scores above the diagonal to minus infinity. Since e to the minus infinity is zero, those weights vanish, and every row still sums to one over the allowed past.

• Cannot peek at future tokens
• Add minus infinity above diagonal

An encoder uses no causal mask, so every token sees both left and right context. This suits understanding tasks such as classification and search, and BERT is the classic example. It is trained by masking out words and predicting them, which makes it poor at free-form generation.

• Bidirectional attention
• BERT-style: understanding tasks

A decoder-only model uses the causal mask and generates one token at a time; most of today's chat models follow this design. The original transformer paired an encoder with a decoder linked by cross-attention, where decoder queries look at encoder keys and values. That layout suits translation and is still used for some tasks.

• Decoder-only: GPT-style
• Encoder-decoder: translation

The training task is disarmingly simple: predict the next token. At the final layer each position outputs a score for every vocabulary entry, and softmax makes it a probability distribution. Given 'the cat sat on the', the model should put high probability on 'mat'. Because of the causal mask, one sentence gives a training example at every position at once.

• Predict token t+1 from tokens up to t
• Softmax over the vocabulary

The loss is cross-entropy, from episode ten. It is simply minus the log of the probability assigned to the true token. If the model gave 'mat' probability 0.5, the loss is about 0.69. If it gave only 0.01, the loss is about 4.6. Confident wrong answers are punished hard, and gradient descent nudges all the weights to lower it.

• Loss = −log p(correct token)
• p = 0.5 → 0.69; p = 0.01 → 4.6

To predict the next word well across enormous amounts of text, a model benefits from learning grammar, facts, style, and even patterns in code and reasoning. This is self-supervised learning: the text itself provides the labels, so no human annotation is needed for pretraining, and that is what lets it scale.

• Grammar, facts, style, code
• Self-supervised: labels are free

Training uses the optimizers we met earlier, typically Adam with weight decay, plus a learning rate that warms up then decays. Because every position is computed in parallel, transformers map beautifully onto GPUs, which is a main reason they displaced recurrent networks. Researchers have also observed that loss tends to fall smoothly as model size, data and compute grow.

• AdamW, warmup, decay
• Many GPUs in parallel

After pretraining we have a base model. It is a powerful text continuer, but ask it a question and it may continue with more questions, because that is what the internet often does next. It has knowledge, yet it has not been taught to behave like an assistant.

• Great at continuing text
• Not yet a helpful assistant

Fine-tuning continues training on a smaller, curated dataset, for example pairs of instructions and good answers. Many assistants then go through a further stage that uses human or automated preference feedback to favor better responses. The base knowledge stays; the behavior changes.

• Supervised fine-tuning on examples
• Preference tuning from feedback

Updating every weight is expensive. For each parameter you must store the weight, its gradient, and optimizer state such as Adam's running averages. For a model with billions of parameters that demands a lot of memory, and storing a full separate copy for every task is wasteful.

• Billions of weights to update
• Weights, gradients, optimizer state

LoRA, low-rank adaptation, freezes the original weight matrix W and learns a small update as the product of two thin matrices: W prime equals W plus B times A. If W is d by d, B is d by r and A is r by d, where r is tiny. The assumption is that useful adaptations need only a low-rank change.

• W' = W + BA
• W frozen; train only B and A

Count the savings. With d equal to 4096, the full matrix has 4096 times 4096, which is 16,777,216 weights. With rank eight, B and A together hold two times 4096 times eight, which is 65,536. That is roughly 0.4 percent. Adapters are small files that can be swapped in and out of one shared base model.

• d = 4096: 16,777,216 weights
• r = 8: 65,536 trainable, about 0.4%

At inference the model produces a distribution over the next token, we choose one, append it, and repeat. Choosing the top token is greedy and repetitive. Sampling adds variety, and temperature divides the scores before softmax: low temperature sharpens the choice, high temperature flattens it.

• Sample from the distribution
• Temperature controls randomness

Attention compares every token with every other, so the score matrix has n squared entries. At one thousand tokens that is a million scores. Double the length to two thousand and it is four million, four times the work. This is why long context windows are costly, and why researchers keep inventing more efficient attention variants.

• n tokens → n² scores
• 1,000 → 1M; 2,000 → 4M

When generating, past tokens do not change, so recomputing their keys and values each step is wasteful. The KV cache stores them. Each new token only computes its own query, key and value, and attends to the cached ones. This saves compute, but the cache uses memory that grows with context length and model depth.

• Cache keys and values of past tokens
• Memory grows with context

Now the honest limits. A model is trained to produce plausible text, not guaranteed true text, so it can state falsehoods fluently, which we call hallucination. It inherits biases from its data, has a knowledge cutoff, and can fail at multi-step reasoning, arithmetic, and counting. Its context is finite, and it cannot verify claims unless connected to tools.

• Hallucination
• Biased or stale knowledge
• Unreliable reasoning

The practical lesson is to treat a language model as a powerful but fallible component. Ground it with retrieved documents, give it tools like calculators and code execution, and verify anything important. Understanding the mechanics we just built tells you where it is strong and where to be careful.

• Verify important claims
• Retrieval and tools help

Let us recap. Text becomes tokens, tokens become embeddings, and positions are added. Attention uses queries, keys and values, scaled by root d k so softmax stays healthy. Multiple heads, residuals, layer norm and feedforward layers make a block, and the causal mask lets us train by next-token prediction. Fine-tuning and LoRA adapt the model cheaply, while inference cost and hallucination remain real constraints.

• Tokens → embeddings + positions
• Attention, heads, residual, norm, FFN

In the final episode we turn to reinforcement learning, where agents learn by trial and reward, then cover advanced topics, putting models into production with MLOps, and capstone projects.

• Episode 30: RL, advanced topics, MLOps
• Capstone projects