Refreshers

14 short, skippable explainers on the concepts my reviews lean on, plus capsules summarizing the papers and posts they build on. Each one is embedded where a review needs it, and lives here too.

Concepts

Attention, model families, and what the KV cache actually stores Queries, keys, and values as a library lookup; who can see whom in encoder, decoder, and encoder-decoder models; and how multi-head, grouped-query, and multi-query attention change the size of the KV cache. 10 min read
Used in Prefill, Decode, and the Quiet Third Job: Reviewing vLLM's Guide to Disaggregated Serving
Compaction, offloading, and retrieval The three classic ways agents keep a growing context under budget, how each one works, and how each one fails. 9 min read
Used in Context Language Models, Dissected: Letting the Model Edit Its Own Context, the Math Behind It, and Where I Disagree
How a multimodal model reads an image Patches, a vision transformer, a projector, and visual tokens: how an image becomes part of a language model's prompt, and what that costs. 7 min read
Used in Prefill, Decode, and the Quiet Third Job: Reviewing vLLM's Guide to Disaggregated Serving
KV caching and prefix caching Why appending to a context is cheap and editing its middle is expensive. 8 min read
Used in Context Language Models, Dissected: Letting the Model Edit Its Own Context, the Math Behind It, and Where I Disagree · Prefill, Decode, and the Quiet Third Job: Reviewing vLLM's Guide to Disaggregated Serving
Mixture of experts: routing, expert parallelism, and why attention and experts behave differently What an MoE layer is, total vs active parameters, how tokens travel to experts across GPUs, and how many tokens each expert actually sees per step. 9 min read
Used in Prefill, Decode, and the Quiet Third Job: Reviewing vLLM's Guide to Disaggregated Serving
Multi-turn conversations and KV reuse Why every chat turn resends the whole conversation, how a single server avoids recomputing it, and what changes when prefill and decode are separate machines. 6 min read
Used in Prefill, Decode, and the Quiet Third Job: Reviewing vLLM's Guide to Disaggregated Serving
Policy gradients to GRPO From REINFORCE to group-relative advantages, and how it extends to multi-call agent trajectories. 12 min read
Used in Context Language Models, Dissected: Letting the Model Edit Its Own Context, the Math Behind It, and Where I Disagree
Reasoning models: thinking tokens, parsers, and what the next turn sees What makes a model a 'reasoning' model, how its thinking is separated from its answer, and why chat templates often drop old thinking from the conversation. 7 min read
Used in Prefill, Decode, and the Quiet Third Job: Reviewing vLLM's Guide to Disaggregated Serving
Tensor, pipeline, data, and expert parallelism in one page The four ways to spread a model across GPUs, what each one splits, what it costs in communication, and what it's good for in serving. 6 min read
Used in Prefill, Decode, and the Quiet Third Job: Reviewing vLLM's Guide to Disaggregated Serving
The agent loop and message lists What an agent's context actually is, who owns it, and why it only ever grows. 5 min read
Used in Context Language Models, Dissected: Letting the Model Edit Its Own Context, the Math Behind It, and Where I Disagree
The Bitter Lesson Sutton's 2019 argument that general methods riding on compute beat built-in human knowledge, plus how to judge a paper that invokes it. 8 min read
Used in Context Language Models, Dissected: Letting the Model Edit Its Own Context, the Math Behind It, and Where I Disagree
TTFT, ITL, TPOT, percentiles, and goodput The handful of numbers every LLM serving benchmark reports, what each one measures from a user's point of view, and why averages hide what users feel. 6 min read
Used in Prefill, Decode, and the Quiet Third Job: Reviewing vLLM's Guide to Disaggregated Serving

Capsules