Reasoning models: thinking tokens, parsers, and what the next turn sees
What makes a model a 'reasoning' model, how its thinking is separated from its answer, and why chat templates often drop old thinking from the conversation.
Skip if You know what a <think> block is, what a reasoning parser does, and whether your model's chat template keeps or strips past reasoning.
Thinking before answering
A reasoning model (DeepSeek-R1, Qwen3 in thinking mode, and many others) is trained to generate an explicit chain of thought before its answer. Reinforcement learning with outcome rewards, such as GRPO (see the policy-gradients and DeepSeekMath refreshers), taught these models that writing out intermediate steps leads to more correct answers. The thinking is just more generated tokens, wrapped in delimiters, often <think> and </think>:
<think>The user wants 17 x 23. 17 x 20 = 340, 17 x 3 = 51, total 391.</think>
17 x 23 = 391.Thinking traces are often long, hundreds to thousands of tokens, which is why reasoning models are expensive to serve: every thinking token is a decode step.
Separating thinking from answer
The model produces one stream of tokens; the API returns structured fields. A reasoning parser on the server finds the delimiters and splits the output into reasoning and content (this is part of the “third job” CPU work). Streaming makes the parser stateful: it has to track whether it’s currently inside a thinking block as chunks arrive.
What happens to old thinking on the next turn?
When the client sends turn two, the conversation history contains turn one’s assistant message. The model’s chat template decides how that message is turned back into tokens, and many templates drop the thinking from past assistant turns, keeping only the final answer. Qwen3’s template does this on its own. The reasons are practical:
- Context budget. Old reasoning is long, and it’s rarely needed once the answer exists.
- Training match. Models were trained on conversations formatted that way.
Some models and agent setups deliberately keep reasoning across turns instead, for example so an agent can keep its plan between tool calls; vLLM documents this under interleaved thinking.
Why it matters for serving
Stripping old thinking means the next prompt is not a continuation of what the model generated: tokens disappear from the middle of the sequence. Anything that reused KV computed for the original sequence now faces a sequence with a hole in it. For a prefix cache that checks block contents, that’s a cache miss after the hole. For anything that reuses blocks by position, it’s a correctness problem, which is the warning in the main post.