Why does an AI answer differently every time? Temperature is the dial behind it.
Hook
Same Prompt, Different Answers?
Why does an AI give a different answer every time you ask?
- Why the variation?
Logits
Words Get Scores
Because it scores every possible word first. Those scores are called logits.
- Every word is scored
- Scores are called logits
Example
Five Candidate Words
Imagine five candidates for the next word: cat, dog, bird, fish, and rock. Their logits are four, three, two, one, and zero.
- cat 4, dog 3, bird 2
- fish 1, rock 0
Softmax
Divide by Temperature
Softmax turns scores into probabilities. But first, divide every logit by the temperature.
- p = softmax(logits / T)
Low temperature
Low T Sharpens
At temperature one, cat gets about sixty-four percent. At zero point five, the gaps double, and cat jumps to about eighty-six percent. The curve sharpens.
- T=1: cat 64%
- T=0.5: cat 86%
High temperature
High T Flattens
At temperature two, the gaps shrink. Cat drops to forty-three percent, and rock climbs to six. The curve flattens.
- T=2: cat 43%
- rock rises to 6%
Takeaway
Pick the Right Dial
So use low temperature for extraction, where you want the same answer every time. Go higher for creative writing.
- Low T: extraction
- High T: creativity
Outro
Next: Full Sampling Lesson
Top-k and top-p are in the full sampling lesson. Follow to catch it.
- Top-k and top-p
- Follow for more