Why does an LLM generate text one word at a time, and why does it get slower as the conversation grows? The answer is the KV cache.
Hook
Why is decoding slow?
Why does an AI write one word at a time, and slow down as the chat gets longer?
- One token at a time
- Slower as chats grow
Analogy
Re-read the whole book
Imagine writing a book. For every new word, you re-read the entire book from page one.
- Every new word
- Start from page one
Keep notes instead
That is wasteful. Smarter writers keep notes, and only read the notes.
- Store, don't recompute
- Notes are the cache
Mechanism
Keys and values saved
In a transformer, those notes are keys and values. For each past token, the model saves them once, then reuses them.
- Store K and V per token
- Reuse at every step
Growth
The cache keeps growing
But the cache grows with every token. Longer context means more memory, for every layer of the model.
- Linear in context length
- Repeated for every layer
Bottleneck
Lots of reading, little math
Here is the catch. Each new token must read that whole cache from memory, but does very little math with it.
- Read all K and V
- Few operations per byte
Memory-bound decoding
So the speed limit is memory bandwidth, how fast data moves, not how fast the chip can calculate.
- Bandwidth is the limit
- Not raw compute
Outro
Full KV cache lesson soon
Full lesson coming soon, with the math, the memory formula, and code. Follow so you don't miss it.
- Math, memory, code
- Follow for more