How GPT Is Trained: Next-Token Prediction Explained with Math and Code

Every large language model you have ever used was built by playing one game, billions of times: guess the next piece of text. No grammar rules, no hand-written facts, no explicit reasoning module. Just a guess, a correction, and a tiny nudge to the weights.

• Guess the next token
• Correct, nudge, repeat

Welcome to the first lesson of Generative AI and LLM Engineering, the follow-on to our Machine Learning from Zero to Advanced series. We will lean on the ML course's lessons on cross-entropy and the Transformer, and then go much deeper into how a model like GPT actually learns.

• Builds on ML course
• Cross-entropy and Transformer recalled

By the end you will be able to define tokens, logits and softmax, derive the training loss, compute it by hand, explain why one sentence gives hundreds of training signals, describe how training data is mixed, and run a tiny GPT-style training loop yourself.

• Derive and compute the loss
• Understand data mixtures
• Run a training loop

Here is the intuition. Think of the autocomplete on your phone, but trained on a huge slice of the internet instead of your last few messages. To predict well, it cannot just memorize phrases. It has to pick up spelling, grammar, facts, code syntax, and even patterns of reasoning, because all of them help guess what comes next.

• Phone autocomplete, but huge
• Good prediction needs understanding

Consider the text: the capital of France is blank. To put high probability on Paris, the model needs a fact. Consider: two plus two equals blank. It needs a little arithmetic. Predicting the next token well forces the network to compress the world described in its training text.

• Facts help prediction
• Skills help prediction

Now the vocabulary. A token is a chunk of text from a fixed vocabulary, produced by the tokenizer from the ML course. It might be a whole word, a word piece, a single character, or a byte. A typical modern vocabulary has tens of thousands to a few hundred thousand entries, and each token has an integer id.

• Chunks from a fixed vocabulary
• Each token has an integer id

The context is the sequence of tokens the model can see when it makes a prediction. Write it as x one, x two, up to x t minus one. The model reads that context and outputs a vector of raw scores, one for every token in the vocabulary. Those raw scores are called logits.

• Context: tokens seen so far
• Logits: one raw score per token

Logits can be any real number, positive or negative, so we convert them to probabilities with softmax. Exponentiate each logit, then divide by the sum of all the exponentials. The results are positive and add up to one. A higher logit always means a higher probability, and the model's whole opinion lives in that distribution.

• Exponentiate, then normalize
• Positive and sums to one

Next, how do we assign a probability to an entire sequence? Use the chain rule of probability. The probability of the whole text equals the probability of the first token, times the probability of the second given the first, times the third given the first two, and so on, all the way to the end.

• p(x) = product of p(x_t | x_<t)
• Exact, not an approximation

Notice this is not a modeling assumption. It is an exact identity that holds for any distribution over sequences. The modeling choice is to use one neural network, with shared weights, to represent every one of those conditional distributions. That is what autoregressive means: each token is predicted from the tokens before it.

• Identity, not assumption
• Shared weights for all positions

Why bother with this form? Because it gives a clean training target and a clean way to generate. To train, we measure how much probability the model gave the real text. To generate, we sample one token, append it to the context, and repeat. We will cover sampling in a later lesson, so today we focus on training.

• Training scores real text
• Generation is covered later

We want parameters theta that make the real training text as probable as possible. That is maximum likelihood. Because probabilities of long sequences are tiny products, we take logs, which turn the product into a sum. And since optimizers minimize, we flip the sign.

• Maximize probability of real text
• Log turns product into sum
• Flip sign to minimize

The result is the next-token cross-entropy loss. At each position, the target is a one-hot vector: probability one on the true next token, zero elsewhere. The cross-entropy between that one-hot target and the model's distribution collapses to a single term, minus the log of the probability the model gave the true token.

• Target is one-hot
• Loss = minus log p(true token)

Look at what this penalty does. If the model gives the true token probability one, the loss is zero. At probability one half, the loss is about zero point six nine. At one tenth, it is about two point three. As the probability approaches zero, the loss shoots toward infinity. Confident mistakes are punished hardest.

• p = 1 gives zero loss
• p = 0.5 gives 0.69
• p = 0.1 gives 2.30

Loss values are easier to feel as perplexity, which is the exponential of the average loss. If the loss is measured in natural logs, perplexity is e raised to the loss. It can be read as the effective number of equally likely choices the model is torn between at each step. A perplexity of four means it is as unsure as picking among four options.

• Perplexity = exp(average loss)
• Effective number of choices

A quick note on units. With natural logs the loss is in nats. Divide by the natural log of two and you get bits per token. Papers use both, so always check which one you are reading. Also remember that losses from different tokenizers are not directly comparable, because the same text splits into a different number of tokens.

• Nats versus bits
• Compare only with same tokenizer

Let us compute it by hand. Use a tiny vocabulary of four tokens: the, cat, sat, and mat. After the context the, suppose the model outputs logits two, one, zero point one, and minus one. The true next token in the training text is cat.

• Vocabulary: the, cat, sat, mat
• Logits: 2.0, 1.0, 0.1, -1.0
• True token: cat

Step one, exponentiate. E to the two is about seven point three nine. E to the one is two point seven two. E to the zero point one is one point one one. E to the minus one is zero point three seven. The sum is about eleven point five eight.

• exp values: 7.39, 2.72, 1.11, 0.37
• Sum is about 11.58

Step two, divide each by the sum. The gets about zero point six four. Cat gets zero point two three five. Sat gets zero point zero nine five. Mat gets zero point zero three two. These add to one. Notice the model liked the most, but the truth was cat.

• the 0.638, cat 0.235
• sat 0.095, mat 0.032
• Model favoured the wrong token

Step three, the loss is minus the natural log of the probability of cat, which is minus the log of zero point two three five. That equals about one point four five nats. The perplexity is e to the one point four five, about four point three, which is also just one divided by zero point two three five.

• Loss = -ln(0.235) = 1.45
• Perplexity about 4.3
• Uniform guess would score 1.39

Now average over several positions. Say a second position gave the true token sat a probability of one half, a loss of zero point six nine. The average of one point four five and zero point six nine is about one point zero seven, so perplexity is about two point nine. And the gradient at the logits is simply the predicted probabilities minus the one-hot target, which pushes cat up and the down.

• Mean loss 1.07, perplexity 2.9
• Gradient = probabilities minus target

Here is the trick that makes training efficient. Take one sequence of, say, a thousand tokens. At position one the target is token two. At position two the target is token three. Every position is its own prediction problem, so one sequence yields nearly a thousand training signals in a single forward pass.

• Each position predicts the next token
• One pass, about T training signals

This is called teacher forcing. During training we always feed the model the true previous tokens, not its own earlier guesses. Even if the model would have predicted a wrong word at position three, position four still sees the correct context. That keeps all the positions independent, and fully parallel.

• Feed true tokens, not guesses
• Positions train in parallel

But there is a catch. If the whole sequence enters the Transformer at once, position three could simply peek at token four, which is its answer. Causal masking prevents this. In the attention lesson we saw scores between every pair of tokens. Here we set the scores for future positions to minus infinity before softmax, so their weights become exactly zero.

• No peeking at future tokens
• Future scores set to minus infinity

Thanks to the mask, position t sees only tokens up to t, exactly as it would at generation time. So training and inference match, while training stays massively parallel on GPUs. One forward pass produces logits at every position, we compute cross-entropy at each, average, and take one gradient step.

• Logits at every position
• Average loss, one gradient step

A model is only as good as the text it predicts. Pretraining data typically comes from several sources: filtered web crawl, which is usually the biggest, plus code, books, encyclopedias, scientific papers, and forums. The exact mixtures of frontier models are mostly undisclosed, so be wary of any precise percentage you see quoted.

• Web, code, books, papers
• Frontier mixtures mostly undisclosed

Raw web crawl is messy, so data goes through a pipeline. First, extract text from HTML. Then filter out low-quality pages, spam, and unwanted content, using heuristics or small classifier models. Then remove personal and unsafe material as policy requires. Each step trades coverage against quality, and teams tune it by experiment.

• Extract, filter, clean
• Quality versus coverage trade-off

Deduplication matters a lot. The web contains the same text over and over, from boilerplate to mirrored pages. Repeated text wastes compute, and the model can memorize it, which raises privacy and copyright concerns. It also contaminates evaluation if benchmark questions leak into training. Teams dedupe exactly, and approximately using techniques like hashing of overlapping text pieces.

• Repeated text wastes compute
• Raises memorization risk
• Exact and near-duplicate removal

Then come the mixture weights: how often each source is sampled. You might upweight books and code because they are dense with structure, and downweight huge but noisy web text. Small high-quality sources may be seen for multiple epochs, meaning several passes, while the giant web portion is seen about once. Too many repeats of the same data eventually stops helping and encourages memorization.

• Sampling weights per source
• Small clean data may repeat
• Too many epochs invites memorization

Finally, data quality has real concerns. Web text carries bias, errors, and outdated claims. Code teaches structure, and may help with logical tasks, but evidence for any single effect varies. Licensing and consent are live debates. And since text on the web increasingly includes model-written text, there is a growing question about how to keep training data diverse and trustworthy.

• Bias, errors, stale facts
• Licensing and consent debates
• Machine-written text in the crawl

Now let us build it. We use a character-level corpus so the vocabulary is tiny. Take a text string, build the sorted set of unique characters, and map each to an integer id. Then encode the whole text as a tensor of ids. For a short poem, the vocabulary might be only thirty or so characters.

• Characters as tokens
• Vocabulary built from the text

Next, batches. Sample random start positions, take a block of, say, sixty-four characters as the input x, and take the same block shifted by exactly one position as the target y. In code, y is data from i plus one to i plus block size plus one. That single shift is the whole idea of next-token prediction.

• x is a block of ids
• y is the same block shifted by one

For the model, we can use an embedding table followed by a small causal Transformer, or even just a single linear layer to start. What matters is the output shape: batch, time, vocabulary. Then we flatten the batch and time axes and call cross-entropy with the targets. PyTorch applies log softmax internally, so we pass raw logits, never probabilities.

• Output shape: batch, time, vocab
• Flatten, then cross_entropy
• Pass raw logits

The loop is the usual one: zero gradients, compute loss, backward, optimizer step, with AdamW. Every few hundred steps, print the loss and its exponential as perplexity. At the start, the loss should be near the natural log of the vocabulary size, about three point four for thirty characters, which is the uniform guess. Then it falls steadily as the model learns spelling and word shapes.

• Start near ln V
• Loss and perplexity fall
• Print loss and exp of loss

Mistake one: confusing loss with accuracy. Suppose two models both rank the correct token first. One gives it probability zero point nine, a loss of about zero point one. The other gives it zero point three, a loss of about one point two. Accuracy is identical, loss is twelve times different. Loss rewards calibrated probabilities, not just right answers.

• Same top-1 hit, different loss
• 0.9 gives 0.11, 0.3 gives 1.20

Mistake two: the off-by-one label shift. If you forget to shift, the model is asked to predict the very token it just read, and the loss collapses to near zero because copying is trivial. If you shift by two, you train a different, harder task. A suspiciously low early loss is the classic symptom. Also, with a real Transformer, check that your causal mask is on, or the model will peek.

• No shift means trivial copying
• Suspiciously low loss is a red flag
• Verify the causal mask

One last piece of context. Everything so far is pretraining, which produces a base model that continues text. A chat assistant then goes through post-training, such as supervised instruction tuning and preference-based methods, which we will preview only here and cover properly in the fine-tuning lessons. Pretraining is where almost all the raw knowledge comes from.

• Pretraining builds the base model
• Post-training shapes behaviour
• Covered later in the course

Recap. A GPT model plays one game: predict the next token. Text is split into tokens, the network outputs logits, and softmax turns them into probabilities. The chain rule factorizes sequence probability, and the loss is the average of minus log probability of the true next token, with perplexity as its exponential.

• Tokens, logits, softmax
• Loss: minus log p of true token
• Perplexity: exp of loss

Teacher forcing and causal masking turn each sequence into many parallel training signals. Data comes from a filtered, deduplicated, weighted mixture, whose exact recipe at the frontier is mostly secret. Watch out for loss versus accuracy, and for off-by-one label shifts. Pretraining builds the base model, and post-training comes later.

• Teacher forcing and masking
• Filtered, deduplicated mixtures
• Avoid classic mistakes

In the next lesson, we ask a big question: why do bigger models trained on more data keep getting better, and can we predict by how much? That is scaling laws.

• Why bigger keeps getting better
• Subscribe for the full series