Serving LLMs with vLLM: Continuous Batching and PagedAttention Explained

Take one GPU and one model. Serve it with a simple loop, and it handles a handful of users. Serve the very same model with vLLM, and the same card can often handle many times more traffic. No new hardware, no new model. The difference comes from two ideas: continuous batching, and PagedAttention.

• One GPU
• Two big ideas
• Continuous batching + PagedAttention

Last episode we shrank models with INT8 and INT4 quantization, so weights take less memory. Today we attack the other half of the problem: how requests are scheduled, and how the KV cache is stored. As in the ML course's lesson on the KV cache, decoding is memory-bound, and that is exactly why this is a systems problem.

• Last time: quantization
• Today: scheduling and memory
• Serving is a systems problem

Here is the plan. First, why naive batching wastes compute and why the KV cache fragments memory. Then continuous batching with a worked timeline. Then PagedAttention with real arithmetic, including prefix sharing. We finish with other vLLM features, runnable Python, tuning knobs, and the mistakes that bite people in production.

• Why naive serving wastes GPU
• Continuous batching
• PagedAttention
• Code and tuning

Decoding one token requires reading all the model weights from GPU memory. That read is the expensive part, not the arithmetic. If we process eight sequences together, we pay for one read of the weights and produce eight tokens. So batching raises throughput almost for free, as long as memory holds out.

• Decode reads all weights each step
• Batching shares that read
• More tokens per second

The simplest approach is static batching. Collect a group of requests, run them together, and return results only when the whole batch finishes. The trouble is that requests produce different numbers of tokens. A request that stops early cannot leave, and its slot sits idle until the longest request in the batch is done.

• Group requests, run together
• Batch ends when the longest ends
• Short requests wait

Let us count. Four requests generate 10, 40, 20 and 80 tokens. The batch runs for 80 steps across 4 slots, so 320 slot-steps are paid for. Useful work is 10 plus 40 plus 20 plus 80, which is 150. Utilization is 150 over 320, about 47 percent. More than half the slot time is idle.

• Lengths: 10, 40, 20, 80
• Batch runs 80 steps
• Useful 150 of 320 slot-steps

The second problem is memory. For each token we store a key and a value in every layer. For a 7 billion parameter class model with 32 layers, hidden size 4096 and 16-bit values, that is 2 times 32 times 4096 times 2 bytes, which is 524,288 bytes. About half a megabyte per token. A 4096 token sequence needs roughly 2 gigabytes.

• Per token: 2 x layers x hidden x bytes
• 7B-class example: 0.5 MB per token
• 4096 tokens is about 2 GB

We do not know how long a response will be in advance. A naive server reserves one contiguous chunk big enough for the maximum length. Suppose the maximum is 2048 tokens, which is 1 gigabyte per request, but a request really uses 300 tokens, 150 megabytes. The other 850 megabytes are locked and empty. Gaps between chunks add external fragmentation on top.

• Unknown output length
• Reserve max length up front
• Internal and external fragmentation

Wasted memory is not just ugly, it limits batch size. If a card has 40 gigabytes free for cache and each request reserves 1 gigabyte, only 40 requests fit, even if their real usage is 6 gigabytes in total. Smaller batches mean fewer tokens per weight read. Memory waste turns directly into lost throughput.

• Wasted memory means fewer sequences
• Fewer sequences means smaller batches
• Lower throughput

Continuous batching, also called iteration-level scheduling, changes the unit of scheduling. Instead of deciding once per batch, the scheduler decides at every single decode step. When a sequence emits its end token, it leaves right away and its slot is freed. A waiting request can enter on the very next step.

• Decide at every decode step
• Finished sequences leave immediately
• Waiting requests join immediately

Redo our example. Requests A to D need 10, 40, 20 and 80 tokens, and two more, E and F, are waiting. At step 10, A finishes and E takes its slot immediately. At step 20, C finishes and F takes that slot. Instead of idle grey, the slots stay full, so the GPU keeps producing useful tokens.

• Same 4 requests plus queued E, F
• A ends at step 10, E joins
• C ends at step 20, F joins

A new request first needs prefill: the whole prompt is processed in parallel to build its KV cache. After that it enters decode, generating one token per step. Prefill is compute-heavy, decode is memory-bound. In continuous batching, a step can contain both new prompts being prefilled and old sequences decoding.

• Prefill: whole prompt in parallel
• Decode: one token per step
• Both share the same steps

Inside the engine there is a waiting queue and a running set. Each step, the scheduler asks: is there enough free cache memory to admit the next request? If yes, it joins. If memory runs short mid-flight, the scheduler can preempt a sequence, pausing it and freeing its cache so it can be recomputed or resumed later.

• Waiting queue and running set
• Admit if memory allows
• Preempt if memory runs out

Continuous batching helps latency, not only throughput. In static batching, a 10 token reply may wait for an 80 token reply. In continuous batching it returns after about 10 steps. New arrivals no longer wait for the whole batch to drain. Under load, the time spent queued is often the biggest part of what users feel.

• Short requests exit early
• New requests start sooner
• Queueing time drops

But continuous batching exposes the next wall. How many sequences can run together is capped by KV cache memory. If we keep reserving contiguous maximum-length chunks, we throw that memory away. To run bigger batches, we need to allocate the cache far more cleverly. That is where PagedAttention comes in.

• Batch size capped by KV memory
• Contiguous reservation wastes it
• Need a better allocator

Operating systems solved this decades ago. A program sees one contiguous virtual address space, but the physical memory behind it is split into fixed-size pages scattered anywhere, and a page table maps one to the other. PagedAttention applies the same trick to the KV cache of each sequence.

• Virtual memory uses pages
• Contiguous virtual, scattered physical
• Page table maps them

vLLM splits the cache into fixed-size blocks, for example 16 tokens each. A sequence gets blocks only as it grows, and each sequence has a block table listing which physical blocks hold its logical blocks. Blocks need not be neighbors. A new block is grabbed from the free list only when the current one fills up.

• Block: 16 tokens of K and V
• Allocated on demand
• Block table per sequence

Back to the 300 token request with maximum length 2048. Naive reservation holds 2048 slots, wasting 1748. With 16 token blocks, we need 300 divided by 16, rounded up, which is 19 blocks, or 304 slots. Waste is just 4 tokens, at most 15 per sequence. At half a megabyte per token, that saves nearly 900 megabytes for this one request.

• 300 tokens, block size 16
• 19 blocks = 304 slots
• Waste: 4 tokens, not 1748

Now apply that to the whole card. Remember the 40 gigabytes that fit only 40 requests when reserving 1 gigabyte each. If each request really uses around 150 megabytes, memory holds many times more of them at once. The real count depends on your traffic, but the direction is clear: usage tracks reality, so batches grow and throughput follows.

• Memory now tracks real usage
• More sequences fit
• Larger batches, more throughput

A fair question: attention needs all previous keys and values. If they are scattered, how does it work? The PagedAttention kernel takes the block table, fetches each block from its physical location, and computes scores block by block. The mathematics is identical to ordinary attention, softmax of query times keys over the whole history. Only the memory addressing changed.

• Custom kernel reads block table
• Gathers K and V per block
• Same math as normal attention

Watch a sequence live. Its prompt fills some blocks. Each decode step appends one token into the last block. When that block hits 16 tokens, the allocator hands over one more free block and the table gets a new entry. When the sequence finishes, all its blocks return to the free pool at once, ready for the next arrival.

• Append token to last block
• Block full: allocate a new one
• Finished: free all blocks

Many requests begin with the same text: a long system prompt, few-shot examples, or a shared document. Identical prefix tokens produce identical keys and values. So instead of storing copies, several block tables can point to the same physical blocks. Suppose a 160 token system prompt, which is 10 blocks, and 8 requests.

• Same system prompt, many requests
• Identical prefix means identical KV
• Point tables to the same blocks

Without sharing, the prefix costs 8 times 10, which is 80 blocks. With sharing it costs 10 blocks, one copy, plus a reference count of 8. We saved 70 blocks, about 1120 tokens of cache, which is roughly 560 megabytes at our half megabyte per token. Prefix caching can also skip recomputing that prefix on later requests.

• Without sharing: 8 x 10 = 80 blocks
• With sharing: 10 blocks
• 70 blocks saved for the prefix

What if a sequence must write into a block that is shared? Then copy-on-write kicks in. If the reference count is above one, the engine copies that block to a fresh one, updates only that sequence's table, and writes there. This is also how sampling several completions from one prompt works: they share the prompt blocks and diverge by copying only the last block.

• Shared block has reference count
• Write to shared block: copy first
• Used for sampling several outputs

A very long prompt can monopolize a step, and every running user sees a stall in their stream. Chunked prefill splits the prompt into pieces and fills each step with a token budget, mixing a chunk of prefill with ongoing decode tokens. The long prompt takes a few steps, but decoding users keep getting tokens smoothly.

• Long prompts hog a step
• Split prefill into chunks
• Mix with decode tokens

Speculative decoding attacks the one-token-per-step limit. A small, fast draft model guesses the next few tokens. The large model then checks all of them in a single parallel pass, accepting the longest correct prefix and fixing the first mistake. Done correctly, the output distribution matches the large model, but it can finish in fewer expensive steps.

• Small draft model guesses k tokens
• Big model verifies in one pass
• Accepted tokens are free

When a model does not fit on one GPU, or you want lower latency, tensor parallelism splits each weight matrix across several GPUs. Each device computes its slice, and they combine results with fast communication every layer. In vLLM you set a tensor parallel size. Interconnect speed matters, so more GPUs is not always faster.

• Model too big for one GPU
• Split weight matrices across GPUs
• GPUs sync each layer

vLLM is not the only choice. TensorRT-LLM from NVIDIA, Hugging Face Text Generation Inference, SGLang, and llama.cpp for local and CPU-friendly use all tackle similar problems, and many share ideas like continuous batching and paged caches. Features and speed change fast, so benchmark on your own model, hardware and traffic.

• TensorRT-LLM
• Hugging Face TGI
• SGLang, llama.cpp

Now the code. For offline batch jobs, import the LLM class from vllm, create it with a model name, and build a SamplingParams object with temperature and max tokens. Then call generate with a list of prompts. vLLM schedules them with continuous batching internally, and returns outputs with the generated text for each prompt.

• LLM class
• SamplingParams
• Batch of prompts in one call

For online serving, run the command vllm serve followed by your model name. It starts an HTTP server that speaks the OpenAI-style chat completions API. Any client library can point its base URL at your server, so switching between a hosted provider and your own GPU is mostly a configuration change.

• vllm serve MODEL_NAME
• Same API shape as chat completions
• Point any client at base_url

Never trust a guess about performance, measure it. Write a client that sends many requests concurrently with asyncio, streams each response, and records the time to first token and total time per request. Then compute output tokens divided by wall-clock time for throughput, plus median and 95th percentile latency. Vary concurrency and watch both curves.

• Fire concurrent requests
• Record time to first token
• Compute tokens per second

Now tuning. The knob max_num_seqs caps how many sequences run in one step. Raise it and throughput can grow, since more tokens share each weight read, but each step takes longer, so per-token latency rises. And it cannot usefully exceed what your KV cache memory can hold. Find the knee of the curve with your load test.

• Max sequences per step
• Higher: more throughput, more latency
• Bounded by KV memory

The setting gpu_memory_utilization is the fraction of GPU memory vLLM is allowed to use. It loads weights, profiles activation needs, and turns whatever remains within that budget into KV cache blocks. A higher fraction means more blocks and more concurrency, but too high risks out-of-memory errors, especially if other processes share the card.

• Fraction of GPU memory vLLM may use
• Weights + activations + KV cache
• Leftover becomes cache blocks

The third knob is max model length, the cap on prompt plus output tokens per request. It decides the worst case a single sequence can need. If one sequence could need 2 gigabytes of cache, the server must be able to hold at least that. Setting it lower than the model's native window, to fit your real traffic, frees room for more concurrent users.

• max_model_len caps prompt + output
• Sets worst case per request
• Lower it to match real traffic

The first common mistake is setting the context length far larger than needed. Maybe one rare request has 100,000 tokens, but you configure everyone for it. The server may fail to start because the cache cannot hold even one maximum sequence, or run with tiny concurrency. Look at your real length distribution, and size for a high percentile, not an extreme outlier.

• Huge max length for rare cases
• Startup fails or concurrency collapses
• Size for the P99 request

The second mistake is ignoring queueing. A load test at low concurrency looks great. But when arrivals exceed what the server can complete, the waiting queue grows without bound, and latency climbs sharply. Track queue depth and time to first token, set request limits and timeouts, and scale out before you hit the cliff, not after.

• Load above capacity grows the queue
• Latency explodes, not gently
• Watch queue depth, set limits

A few more traps. Benchmarking with one fixed prompt hides the real mix of lengths. Quoting throughput without latency hides a poor user experience. And first requests can be slower from startup effects, so warm up before measuring. Finally, report results with your hardware, model, and settings, so numbers are never invented or mysterious.

• Benchmarking with one prompt length
• Comparing tokens/s without latency
• Forgetting to warm up

Let us recap. Static batching wastes slots until the longest request ends; continuous batching schedules every step and refills slots instantly. Contiguous KV reservations waste memory; PagedAttention uses fixed blocks and block tables, enabling bigger batches, prefix sharing and copy-on-write. Add chunked prefill, speculative decoding and tensor parallelism, then tune max_num_seqs, memory utilization and max length by measuring.

• Continuous batching: schedule per step
• PagedAttention: blocks and block tables
• Tune with measurements

Your model is now fast and affordable to serve. But fast does not mean safe. In the next episode we cover guardrails, safety, prompt injection and monitoring in production.

• Next: guardrails and safety
• Prompt injection and monitoring
• Subscribe for the series