Capsule: Inside vLLM, how one engine mixes prefill and decode

#The source

Inside vLLM: Anatomy of a High-Throughput LLM Inference System by Aleksa Gordić (September 2025, originally on his own site) is the best walk-through of vLLM’s V1 engine I know. Its analysis is pinned to a commit from August 2025, so some names may have changed since; the ideas haven’t. Here is the part you need for disaggregation, in my words.

#One engine, one loop

An engine repeatedly calls step(), and each step has three stages:

1. Schedule: choose which requests run in this step, and how many tokens each gets.
2. Forward pass: run the model once over all of those tokens, then sample.
3. Postprocess: append the sampled tokens, detokenize, check stop conditions, and free the KV blocks of finished requests.

Everything that matters for this post happens in stage 1.

#The scheduler’s token budget

The scheduler treats the two kinds of work very differently:

• Decode requests (already running) need one new token each, plus a few more when speculative decoding is on. The scheduler handles these first.
• Prefill requests (waiting) need all of their prompt tokens processed. They go second, into whatever room is left in the step’s token budget.

Since the V1 engine, both kinds can share a single step. The forward pass flattens every request in the batch into one long “super-sequence,” and positions plus attention masks keep each request looking only at its own tokens. That’s continuous batching: requests join and leave between steps, with no padding.

Chunked prefill caps how many prompt tokens one request may take in a step, so a long prompt is processed over several steps instead of monopolizing one. A prompt of three chunks needs at least three steps, and its first output token is sampled only in the last one.

#Paged KV memory

The KV cache is stored in fixed-size blocks, 16 tokens each by default, handed out from a pool of free blocks. For a standard transformer layer, one block occupies

where $B$ is the block size in tokens, $h_{kv}$ the number of KV heads, $d_h$ the head dimension, and $b$ the bytes per value. Each request owns a list of blocks rather than one contiguous region, which is what lets thousands of requests of different lengths share GPU memory without fragmentation. Prefix caching builds on the same blocks: full blocks are hashed (each hash chained to the previous block’s), so a new request whose prompt begins with already-cached blocks reuses them instead of recomputing them.

#Where disaggregated P/D plugs in

Prefill is compute-bound and decode is memory-bound, so the source frames P/D as running $N$ prefill instances and $M$ decode instances, sized to the live request mix, with KV caches flowing from one to the other. The piece that moves them is a KV connector, created twice, once inside the scheduler and once inside each worker. Its lifecycle in a step:

1. Lookup. While scheduling a waiting request, the scheduler asks the connector how many of its tokens already have KV available externally. Prefill always gets 0; decode may get a hit.
2. Bookkeeping. After allocating blocks, the scheduler records which requests will load or save KV, and builds connector metadata for the step.
3. Load before, save after. The forward pass runs inside a context manager. On entry, decode starts loading external KV into its paged blocks; on exit, prefill waits until its KV has been saved.

The source illustrates this with a simple shared-storage connector (a local filesystem standing in for a KV service). Production connectors such as NIXL move the blocks over RDMA or GPU-to-GPU links instead. The lifecycle is the same.

#Why this matters for the post

In a collocated engine, decode and prefill share every step: decodes go first, but if a prefill chunk is admitted, the whole step, decodes included, takes as long as that chunk. That’s the mechanical reason streams stutter when long prompts arrive. Disaggregation removes the prefill chunk from the decode engine’s steps entirely; the connector is the price.