Multi-turn conversations and KV reuse

#Every turn resends everything

Chat APIs are stateless: on turn two, the client sends the system prompt, the first user message, the model’s first answer, and the new message, all of it, again (see the agent-loop refresher). Agents make this extreme. A coding agent can run dozens of turns per session, each one replaying a context that only grows.

#A single server reuses almost all of it

On one vLLM engine, the KV cache for turn one’s prompt and for the answer it generated sit in the same paged memory. With prefix caching on, turn two’s prompt matches all of those blocks by hash (see the KV-caching refresher), so the engine computes only the new message. Time to first token on turn five can be a fraction of turn one’s.

#Disaggregation splits the history across two machines

With separate prefill and decode instances, the history is split:

• Prefill computed turn one’s prompt, so it can cache those blocks.
• Decode generated the answer, so only decode has the answer’s KV.

On turn two, prefill sees a prompt containing an answer it never computed. In standard, one-way P/D, it recomputes that answer, every turn. The longer the model’s answers (reasoning traces, tool calls), the more work that is.

Bidirectional KV transfer closes the loop. Decode keeps its blocks for a while after the turn, and on the next turn prefill pulls the blocks it’s missing back from decode, then computes only the new message. Something has to remember which blocks belong to which conversation; in vLLM, that’s a proxy keyed by a conversation ID the client sends.

#The assumption underneath

Every form of KV reuse assumes the new prompt continues the old sequence: same tokens, same positions, with new tokens only at the end. A single server’s prefix cache checks that assumption block by block, so a mismatch only costs a recompute. Any mechanism that reuses blocks without checking turns a broken assumption into wrong output.