Sampling Explained: Greedy, Temperature, Top-k, Top-p (With Numbers)

Ask a language model the same question twice and you may get two different answers. Turn one setting down and it becomes dry and predictable. Turn it up and it gets creative, or completely unhinged. The weights did not change at all. The only thing that changed is how we pick the next token.

• Same weights
• Different outputs
• One hidden dial

In the last episode we saw why scaling a model keeps lowering its loss. Today we zoom in on the very last step of generation: turning the model's raw scores into one chosen token. We will cover greedy decoding, temperature, top-k, top-p, a few modern tricks, and then implement them all in PyTorch.

• Last step of generation
• Scores become a token
• Then code it ourselves

Here is the key intuition. A language model does not output a word. It outputs a score for every token in its vocabulary, tens of thousands of them. Those scores become a probability distribution. Choosing the actual token is a separate decision, called decoding, and that decision is entirely up to us.

• Model outputs scores
• Scores form a distribution
• Decoding is our choice

The raw scores are called logits, written z. They can be any real number, positive or negative. To turn them into probabilities we use softmax: the probability of token i is e to the power z i, divided by the sum of e to the power z j over all tokens j. Everything is positive and the total is exactly one.

• z: raw logits
• Softmax makes probabilities
• All positive, sum to one

Let us use a tiny example. The prompt is the cat sat on the, and the model considers five candidates: mat, floor, sofa, roof and moon. Their logits are four, three, two, one and zero. Exponentiating gives about fifty-five, twenty, seven point four, two point seven, and one. The sum is about eighty-six.

• Prompt: the cat sat on the
• Logits 4, 3, 2, 1, 0
• Sum of exponentials: 85.8

Dividing by eighty-six, we get probabilities at temperature one: mat is sixty-four percent, floor twenty-three percent, sofa nine percent, roof three percent, and moon just over one percent. Notice that the logits were evenly spaced, but the probabilities are not. Exponentiation rewards the leader disproportionately.

• mat 64%, floor 23%
• sofa 9%, roof 3%, moon 1%
• Even logits, uneven probabilities

Temperature is a single number T that we divide the logits by before softmax. The formula becomes p i equals e to the z i over T, divided by the sum of e to the z j over T. When T is one, nothing changes. The question is what happens when T is smaller or larger than one.

• Divide logits by T
• T = 1 changes nothing
• Smaller or larger T?

Set T to zero point five. Dividing the logits by that is the same as doubling them: eight, six, four, two, zero. Exponentiate: about twenty-nine hundred and eighty, four hundred and three, fifty-five, seven, and one. The total is about thirty-four hundred. Now mat gets eighty-six percent, floor twelve percent, sofa about one point six, and the rest almost nothing.

• Logits double: 8, 6, 4, 2, 0
• mat 86%, floor 12%
• Distribution sharpens

Now set T to two. The logits are halved: two, one point five, one, zero point five, zero. Exponentiating gives seven point four, four point five, two point seven, one point six and one, totalling about seventeen point two. The probabilities become forty-three, twenty-six, sixteen, ten and six percent. Even moon now has a real chance.

• Logits halve: 2, 1.5, 1, 0.5, 0
• mat 43%, moon 6%
• Distribution flattens

We can measure how spread out a distribution is with entropy, the negative sum of p times log base two of p. At T equal to one the entropy is about one point four four bits. At T equal to zero point five it falls to about zero point six six bits. At T equal to two it rises to about two point zero one bits. The maximum for five tokens is two point three two bits.

• T=0.5: 0.66 bits
• T=1: 1.44 bits
• T=2: 2.01 bits, max 2.32

Take the extremes. As T goes to infinity, every logit divided by T approaches zero, so the distribution becomes uniform and the output is random noise. As T goes to zero, the largest logit dominates completely and all the probability piles onto one token. Temperature never changes the ranking of tokens, only how much the leader wins by.

• T to infinity: uniform noise
• T to zero: all mass on top
• Ranking never changes

The simplest decoder is greedy: at every step, take the token with the highest probability, the argmax. It is fast and fully deterministic. In our example it always picks mat. For tasks with one clearly correct answer, that can be exactly what you want.

• Pick the argmax each step
• Fast and deterministic
• Fine when one answer is right

But greedy has a famous failure: repetition loops. Once the model writes a phrase, that phrase makes itself more likely to appear again, so the top token keeps pointing back into the same cycle. Greedy has no way to escape, because it never takes a risk. Text like I do not know, I do not know, I do not know is the classic symptom.

• Phrases reinforce themselves
• No randomness to escape
• Loops and bland text

The fix is to sample instead: draw a token at random, with each token chosen in proportion to its probability. At T equal to one, mat is picked sixty-four percent of the time, floor twenty-three percent, and so on. Over many runs, the text varies, yet common choices still win most of the time. Temperature now becomes a real creativity knob.

• Draw tokens proportionally
• Variety across runs
• Temperature controls the spread

Pure sampling has its own problem. A vocabulary has tens of thousands of tokens, and the tail of improbable ones can add up to a surprisingly large mass. Imagine thirty thousand junk tokens each with a hundredth of a percent. That is three percent in total, so every few dozen tokens you may pick garbage, and one bad token can derail a whole sentence.

• Tail tokens are individually rare
• Together they add up
• One bad token derails text

Top-k sampling cuts the tail by force. Keep only the k most likely tokens, throw away the rest, and renormalize. With k equal to three at T equal to one, we keep mat, floor and sofa with a combined mass of about ninety-six percent. Dividing each by point nine six six gives sixty-seven, twenty-four and nine percent.

• Keep the k best tokens
• Drop the rest
• Renormalize: divide by 0.957

The trouble is that k is fixed, while the model's confidence is not. After the words the capital of France is, one token deserves almost all the mass, and keeping fifty is wasteful. After I feel like, dozens of continuations are fine, and keeping only ten may be too strict. A fixed k cannot adapt to the shape of the distribution.

• Confident step: k too loose
• Open step: k too tight
• k ignores the distribution shape

Top-p, also called nucleus sampling, adapts. Sort tokens from most to least likely, then keep the smallest set whose cumulative probability reaches p. Everything outside that nucleus is dropped, and the survivors are renormalized. The number of tokens kept changes at every step, depending on how confident the model is.

• Sort by probability
• Accumulate until reaching p
• Keep that smallest set

Use p equal to zero point eight at T equal to one. Cumulative sums are zero point six four for mat, then zero point eight seven after floor. That already passes zero point eight, so we stop with two tokens. Renormalizing, mat gets seventy-three percent and floor twenty-seven percent. Sofa, roof and moon can never be chosen.

• Cumulative: 0.64, then 0.87
• Passes 0.8, so keep two
• Renormalize to 73% and 27%

Now watch it adapt. With p equal to zero point nine, the sharp T equal to zero point five distribution needs only two tokens, since the cumulative mass is already ninety-eight percent. At T equal to one we keep three tokens. At T equal to two the cumulative sums are zero point four three, zero point six nine, zero point eight five, zero point nine four, so four tokens survive.

• T=0.5 keeps 2 tokens
• T=1 keeps 3 tokens
• T=2 keeps 4 tokens

In most libraries the pipeline is: divide the logits by temperature, apply top-k, apply top-p, then softmax and sample. Since temperature changes the shape of the distribution, it changes what top-p keeps, as we just saw. That interaction matters. Filters are applied by setting the removed logits to minus infinity, so softmax gives them exactly zero probability.

• Temperature first
• Then top-k, then top-p
• Masked logits become minus infinity

A newer filter is min-p. Instead of a cumulative total, it sets a threshold relative to the top token: keep any token whose probability is at least min-p times the maximum. With min-p of zero point one and a top probability of zero point six four, the cutoff is about zero point zero six. Mat, floor and sofa stay, roof and moon go.

• Threshold is a fraction of the max
• Confident: strict filter
• Uncertain: lenient filter

To fight repetition directly, we can penalize tokens that already appeared. A frequency penalty subtracts a value times the count from the logit. If mat appeared twice and the penalty is one, its logit drops from four to two, and floor takes the lead. A repetition penalty instead scales logits of seen tokens, for example dividing a positive logit by one point two. Use gently, because strong penalties push the model off correct words.

• Frequency: subtract per count
• Repetition: scale seen logits
• Strong penalties hurt accuracy

Beam search is a different idea. Instead of one path, keep the B best partial sentences, ranked by total log probability, and extend them all at every step. It was a staple in translation and summarization, where one faithful output is the goal. For open-ended chat it tends to produce generic, repetitive text, and it costs B times the compute, so chat models rarely use it.

• Keep B best sequences
• Great for translation
• Bland and costly for chat

Now the code, in plain PyTorch. Load a small model such as GPT-2 with the Hugging Face transformers library, tokenize a prompt, and run one forward pass. The logits for the next token are the last position of the output: logits equals model of input ids dot logits, indexed zero, minus one. That vector of vocabulary size is all our samplers need.

• Load a small GPT-2
• One forward pass
• Take logits at last position

Greedy is one line: torch dot argmax of the logits. Temperature is also tiny. Divide the logits by T, call softmax over the last dimension, then torch dot multinomial with one sample. Guard against T equal to zero by falling back to argmax, because dividing by zero would give infinities and not a number errors.

• Greedy: argmax
• Temperature: divide, softmax, sample
• Guard against T equals zero

For top-k, use torch dot topk to find the k-th largest value, and set every logit below it to minus infinity. For top-p, sort the logits in descending order, take the softmax, and compute the cumulative sum. Mark tokens where the cumulative sum minus their own probability already reaches p, since that keeps the first token that crosses the line. Mask those, then scatter back to the original order.

• Top-k: threshold at k-th value
• Top-p: sort and cumulative sum
• Keep the token that crosses p

Let us compare. Using the prompt, once upon a time, generate fifty tokens with each strategy. Greedy typically loops or turns bland. Temperature one with no filter can wander into nonsense. Temperature zero point eight with top-p zero point nine usually reads fluently with variety. Your exact outputs will differ, so run it several times with different seeds, and notice how much diversity each setting gives.

• Greedy: loops or bland
• Pure sampling: drifts
• Moderate T plus top-p: fluent

Match settings to the job. For extraction, classification, code, and anything with one right answer, use temperature at or near zero. For chat, a moderate temperature around zero point seven is a common default. For brainstorming, poetry and story ideas, go higher and let top-p or min-p keep it coherent. These are starting points, not laws, so test on your own data.

• Extraction and code: near zero
• Chat: moderate
• Creative: higher with a filter

A common belief is that temperature zero makes a hosted API perfectly deterministic. Often it does not. Floating point addition is not associative, so different batch sizes, hardware or kernel choices can change the last digits of a logit. When two tokens are nearly tied, that tiny difference flips the argmax, and the text diverges from there. Providers also change models over time.

• Floating point is not associative
• Near ties can flip
• Do not rely on exact repeats

The first mistake is stacking temperature, top-p, top-k and penalties all at once without understanding the interaction. Raising temperature to one point five and then setting top-p to zero point nine flattens the distribution and then keeps a bigger nucleus, so the two fight each other. A good habit is to change one knob at a time, and many providers advise adjusting temperature or top-p, not both.

• Knobs interact
• Change one at a time
• Tune with evals, not vibes

The second mistake is expecting identical outputs from a stochastic process. Even with sampling, you can set a seed locally and reproduce results on the same machine and software. Across hardware, library versions or hosted services, reproducibility is best effort. If your pipeline needs stability, validate outputs, use structured formats, and design tests that tolerate harmless variation instead of comparing strings exactly.

• Seeds help locally
• Hosted APIs: best effort
• Test with tolerance

To recap: the model outputs logits, softmax turns them into probabilities, and temperature divides the logits to sharpen or flatten that distribution. Greedy takes the argmax and can loop. Top-k keeps a fixed number of tokens, top-p keeps an adaptive nucleus, and min-p scales with the leader. Penalties fight repetition, and beam search is rare in chat.

• Logits, softmax, temperature
• Greedy, top-k, top-p, min-p
• Penalties and beam search

In the next episode we ask how much a model can read at once. We will look at context windows and positional encoding, and derive rotary embeddings, or RoPE, step by step.

• Next: context windows and RoPE
• Subscribe for the series
• Try the samplers yourself