How LLMs Actually Run: Decoding, KV Cache, Latency and Limits

You have trained, or at least understood how to train, a language model. Now comes the part every user actually experiences: you type a prompt, and words come back. How fast do they come, how much memory does it cost, and how far can you trust them? That is inference, and it is today's lesson.

• Inference: using the trained model
• Cost, speed, memory, mistakes

In lesson 57 we saw how a model is pretrained, fine-tuned, and adapted cheaply with parameter-efficient methods. The weights are now frozen. In this lesson we ask what happens every time those frozen weights are used: how tokens are chosen, why a cache matters, what limits speed, and where these models fail.

• Lesson 57: pretraining, fine-tuning, PEFT
• Lesson 58: running the model

A language model does not write a paragraph at once. Given all tokens so far, it outputs a probability for every token in the vocabulary as the next one. We choose one, append it to the text, and feed everything back in. This loop is called autoregressive decoding, and every strategy in this lesson is a rule for the choosing step.

• Model predicts next-token distribution
• Pick a token, append, repeat

The network's raw outputs are logits. Suppose four candidate tokens, A, B, C and D, have logits three, two, one and zero. Softmax exponentiates each: about twenty point one, seven point four, two point seven, and one. These add to roughly thirty-one point two. Dividing gives probabilities of about sixty-four, twenty-four, nine, and three percent.

• Logits z, softmax p = e^z / sum e^z
• Four tokens: 3, 2, 1, 0

The simplest rule is greedy decoding: always take the most probable token, here token A. It is fast and deterministic, which is useful for extraction or code. But it can be dull, and it can fall into loops, repeating the same phrase because that phrase keeps being the single most likely continuation. Often we want some controlled randomness.

• Always take the argmax
• Deterministic but repetitive

Temperature rescales the logits before softmax: divide each logit by T. When T is one, nothing changes. When T is below one, differences are magnified; when T is above one, they are shrunk. It is a dial between confident and adventurous, and it adds no new parameters to the model.

• p = softmax(z / T)
• T below 1 sharpens, above 1 flattens

Take T equals zero point five. The logits become six, four, two, and zero. Exponentiating gives about four hundred three, fifty-five, seven, and one, summing to roughly four hundred sixty-six. Token A now has probability eighty-six and a half percent, up from sixty-four. The distribution has sharpened, and as T approaches zero this becomes greedy decoding.

• Logits become 6, 4, 2, 0
• Probabilities 0.865, 0.117, 0.016, 0.002

Now T equals two. The logits halve to one point five, one, point five, and zero. Exponentials are about four point five, two point seven, one point six, and one, summing to nearly ten. Probabilities become forty-five, twenty-eight, seventeen, and ten percent. Token D, nearly impossible before, is now chosen one time in ten. High temperature means variety, and also more nonsense.

• Logits become 1.5, 1, 0.5, 0
• Probabilities 0.455, 0.276, 0.167, 0.102

Pure sampling has a flaw: the long tail of unlikely tokens adds up. Top-k fixes this by keeping only the k most probable tokens. With k equal to two at temperature one, we keep A and B, with total mass zero point eight eight one. Renormalize by dividing: A becomes seventy-three percent, B twenty-seven. Then we sample from just those two.

• Keep only the k best tokens
• Renormalize, then sample

Top-p chooses the smallest set of tokens whose cumulative probability reaches p. With p equal to zero point nine, the running totals are zero point six four four, zero point eight eight one, then zero point nine six eight. We must include A, B and C. Renormalized, they get sixty-six, twenty-four and nine percent. Unlike top-k, the set grows when the model is unsure and shrinks when it is confident.

• Smallest set with cumulative mass at least p
• Set size adapts to the situation

A few common mistakes. Turning every knob at once makes behavior hard to reason about, so change one at a time. Low temperature does not make answers true, only more consistent. And high temperature does not make a model creative in any deep sense, it just flattens probabilities. Sampling controls style and diversity, not correctness.

• Temperature zero is not creativity-safe
• Stacking settings blindly
• Randomness is not truthfulness

Greedy choices are locally best, not globally best. Imagine the first token A has probability zero point six, and B has zero point four. After A, the best next token has only zero point four, so the greedy path scores zero point two four. After B, the best next token has probability zero point nine, giving zero point three six. Greedy never explores B.

• Best first token may lead to a bad path
• We want the best whole sequence

Beam search keeps b partial sequences at every step. With width two, we keep both A and B, expand each, score all children by total probability, and keep the best two. It finds the zero point three six path. In practice we add log probabilities instead of multiplying, to avoid numerical underflow on long sequences.

• Beam width b partial sequences
• Expand all, keep the b best
• Score with sum of log probabilities

Beam search shines when there is a fairly well-defined right answer, such as translation. For open-ended conversation it tends to produce safe, generic, repetitive text, which is why chat systems usually sample instead. Beam search also multiplies the compute and memory by the beam width, and longer sequences need a length correction because every added token lowers the total probability.

• Good for translation, summarization
• Bland, repetitive for open chat
• Cost grows with beam width

Recall from lesson 55 that attention compares each token's query with the keys and values of earlier tokens. If we naively rerun the whole model on the full text at every step, we repeat work. To generate one thousand tokens, we would process one, then two, then three tokens, and so on, a total of five hundred thousand five hundred token passes instead of one thousand.

• Each step reprocesses the whole prefix
• 1000 tokens: 500,500 token passes

The key insight: because of the causal mask, a past token's keys and values never change when new tokens arrive. So we store them. For each new token, we compute only its own query, key and value, append the key and value to the cache, and attend over everything stored. Each step now processes one token, not the whole prefix.

• Past keys and values never change
• Compute only the new token's K, Q, V

Speed costs memory. Take a sample model with thirty-two layers and hidden size four thousand ninety-six, stored in sixteen-bit floats, two bytes each. Per token and per layer we store a key and a value, each of four thousand ninety-six numbers. That is two times thirty-two times four thousand ninety-six times two bytes: five hundred twenty-four thousand, two hundred eighty-eight bytes, half a megabyte per token.

• 32 layers, hidden size 4096, fp16
• 2 x 32 x 4096 x 2 bytes
• About 0.5 MB per token

Now scale it. A single four thousand ninety-six token sequence needs about two gigabytes of cache. Thirty-two such sequences at once need sixty-four gigabytes. Meanwhile, a seven-billion-parameter model in sixteen-bit weights needs about fourteen gigabytes. The cache, not the weights, often becomes the limit on how many users one GPU can serve.

• 4096 tokens: 2 GB per sequence
• 32 sequences: 64 GB
• Weights alone: about 14 GB

Designers fight this growth. Grouped-query attention lets several query heads share one set of keys and values. If our sample model used eight key-value heads instead of thirty-two, the cache would shrink fourfold, to about an eighth of a megabyte per token. Systems can also reuse the cached prefix when many requests share the same system prompt.

• Grouped-query attention shares K and V
• 8 KV heads instead of 32: 4x smaller
• Prefix caching reuses shared prompts

Latency is how long one user waits. Throughput is how many tokens the whole system produces per second. They pull in different directions. Generation has two phases: prefill, where the whole prompt is processed in parallel, determining time to first token, and decode, where tokens are produced one by one, determining the streaming speed.

• Latency: time for one user's answer
• Throughput: tokens per second overall
• Time to first token vs per-token time

For one user, decoding does very little arithmetic per byte moved. Each new token needs every weight read from memory once. With fourteen gigabytes of weights and a hypothetical memory bandwidth of one thousand gigabytes per second, that is fourteen milliseconds per token, around seventy tokens per second at best. The arithmetic units mostly wait for data.

• Every token reads all 14 GB of weights
• Assume 1000 GB/s bandwidth
• About 14 ms per token, about 70 tokens/s

Batching fixes the waste. If thirty-two requests are decoded together, the weights are read once and used for all of them, so throughput can rise close to thirty-two times with little extra latency. Eventually the GPU becomes compute-bound, or the caches fill memory, and each user starts waiting longer. Serving systems add requests to the batch continuously as others finish.

• One weight read serves many requests
• Throughput rises almost linearly at first
• Latency and cache memory eventually suffer

Quantization stores weights with fewer bits. A seven-billion-parameter model takes fourteen gigabytes at two bytes per weight, seven at one byte, and about three and a half at four bits. Each weight is rounded to the nearest of a few levels, using a scale factor. Smaller weights also mean fewer bytes to read, so decoding gets faster.

• fp16: 2 bytes, int8: 1, 4-bit: 0.5
• 7B model: 14 GB, 7 GB, 3.5 GB

The cost is rounding error. If the largest weight in a group is one, and we have 127 positive levels, each step is about zero point zero zero eight. A single outlier stretches the grid and makes ordinary weights coarser, so practical methods quantize in small groups and treat outliers carefully. Eight-bit usually costs little quality; four-bit needs checking on your own tasks.

• Scale = max / levels
• Outlier weights hurt accuracy
• Always measure quality afterwards

Distillation attacks the problem differently. Train a small student model to imitate a large teacher. Instead of only the correct word, the student sees the teacher's full probability distribution, which carries rich information about which alternatives are plausible. We minimize cross-entropy between the two, as in lesson 10. The result is a smaller, cheaper model that keeps much, though not all, of the teacher's ability.

• Teacher gives soft probabilities
• Student minimizes cross-entropy to teacher
• Smaller model, cheaper inference

Every model has a context window: the maximum number of tokens it can consider at once. Your instructions, retrieved documents, chat history and the answer being generated all share that budget. Anything beyond is simply not seen. So long documents must be truncated, summarized, or split.

• Maximum tokens the model can attend to
• Prompt plus output share the same budget

Attention compares every token with every other, producing an n by n score matrix per head and layer. With one thousand tokens that is one million scores. Eight times longer, eight thousand tokens, gives sixty-four million: sixty-four times the work. Doubling the context quadruples the attention cost, which is why long contexts are expensive.

• n tokens give n x n scores
• 1000 tokens: 1 million scores
• 8000 tokens: 64 million scores

Practical tricks help. Fused kernels compute attention in small tiles so the full score matrix is never stored, saving memory, though the arithmetic remains quadratic. The cache grows linearly with length, as we computed. And a model that accepts a long context does not necessarily use it well: information buried in the middle of a long prompt can be missed.

• Memory-efficient attention kernels
• Cache grows linearly, scores grow quadratically
• Long context is not perfect recall

The first limitation is hallucination. The model is trained to produce plausible next tokens, and nothing in the loop checks facts. When it lacks knowledge, it can still generate a confident, fluent, wrong answer, such as an invented citation. The probability of a token measures plausibility under the training data, not truth about the world.

• Fluent text is not verified text
• Model optimizes plausibility, not truth

Next, three data issues. Bias: models absorb patterns, including unfair ones, from their training text. Privacy: models can memorize and occasionally reproduce rare sequences from training data, so sensitive data needs care. Contamination: if benchmark questions appeared in the training data, a high score may reflect memory rather than ability, echoing the leakage lessons from lesson 35.

• Bias mirrors training data
• Models can memorize private text
• Benchmark leakage inflates scores

Models are also sensitive to phrasing: reordering a prompt or changing one word can change the answer. And their reasoning is unreliable. A model can write steps that look convincing, yet contain an error, and the explanation does not always reflect how the answer was produced. For math or logic, use tools such as calculators or code, and check results.

• Rewording changes answers
• Reasoning can look right but be wrong
• Verify arithmetic and logic externally

Retrieval-augmented generation, or RAG, reduces several problems at once. Embed the question, as in lesson 54, search a document store for the closest passages, and place them in the prompt. The model then answers from supplied evidence, which can be current and private, and it can cite its sources. RAG is not magic: poor retrieval or ignored context still yields wrong answers.

• Retrieve relevant documents first
• Put them in the prompt
• Ask the model to answer from them

Finally, evaluation. Collect real examples of your task with trusted answers, keep a held-out portion unseen during prompt tuning, and score outputs for correctness, grounding in sources, and safety. Rerun after each change to catch regressions. Spot checks and human review remain essential, and remember that automated judges, including other models, have biases of their own.

• Build a test set from real tasks
• Measure accuracy, groundedness, safety
• Track regressions after every change

Let us recap. Decoding picks tokens from a softmax; temperature, top-k, top-p and beam search shape that choice. The KV cache trades about half a megabyte per token, in our sample model, for far less recomputation. Batching, quantization and distillation improve cost and speed. Attention cost grows quadratically with context, and LLMs still hallucinate, so retrieval and evaluation are your safeguards.

• Decoding: greedy, temperature, top-k, top-p, beam
• KV cache: speed for memory
• Batching, quantization, distillation
• Limits: context cost and reliability

That completes our tour of how language models run, and where they stop. Next, in lesson 59, we begin reinforcement learning with Markov decision processes and the Bellman equations: learning from rewards instead of labels.

• Lesson 59: MDPs and Bellman equations
• Subscribe for the full series