← All refreshers Refresher · 10 min read

Attention, model families, and what the KV cache actually stores

Queries, keys, and values as a library lookup; who can see whom in encoder, decoder, and encoder-decoder models; and how multi-head, grouped-query, and multi-query attention change the size of the KV cache.

Skip if You can explain why decoder-only models cache keys and values, and compute KV bytes per token from a model config.

Attention as a library lookup

Every token, at every layer, is turned into three vectors:

  • a query (qq): what this token is looking for;
  • a key (kk): a label saying what this token contains;
  • a value (vv): the information this token hands over if it’s chosen.

Think of a library. Your question is the query; the catalogue cards are the keys; the books are the values. You compare your question against every card, decide how relevant each one is, and read a weighted mix of the matching books. In math, for token ii:

outi=∑jwijvjwij=softmax⁡j(qi⋅kj/dh)\begin{aligned} \mathrm{out}_i &= \sum_{j} w_{ij}\, v_j \\ w_{ij} &= \operatorname{softmax}_j\big(q_i \cdot k_j \,/\, \sqrt{d_h}\big) \end{aligned}

Here wijw_{ij} is how relevant card jj is to the question.

Here dhd_h is the head dimension, and the dh\sqrt{d_h} keeps the dot products in a range where the softmax doesn’t saturate. Which jj the sum runs over, meaning which tokens a token is allowed to look at, is what separates the three model families.

Three families, three masks

Attention masks for encoder-only, decoder-only, and encoder-decoder models

  • Encoder-only (BERT, ModernBERT, embedding models). Every token sees every other token, in both directions. The whole input is processed in one pass and nothing is generated token by token, so there’s no “decode” phase at all. Serving these is all prefill.
  • Decoder-only (GPT, Llama, Qwen). Each token sees only itself and the tokens before it: a causal mask, the lower triangle. That’s what makes generation possible: the model predicts token tt from tokens 1…t−11 \dots t-1, appends it, and repeats. These models have both phases: a parallel prefill over the prompt, then step-by-step decode.
  • Encoder-decoder (T5, Whisper). An encoder reads the input with full attention, once. The decoder then generates causally, and each decoder layer also cross-attends to the encoder’s outputs. The keys and values for that cross-attention are computed once from the encoder and reused at every decoding step.

Almost every chat and agent model served today is decoder-only, so the rest of this refresher focuses on that case.

Why decoder-only models cache K and V

Generation is autoregressive: the probability of an answer yy to a prompt xx factors into one prediction per token,

p(y∣x)=∏t=1Tp(yt∣x,y<t),p(y \mid x) = \prod_{t=1}^{T} p\big(y_t \mid x,\, y_{<t}\big),

where y<ty_{<t} means all the answer tokens before tt. So producing TT tokens takes TT forward passes. With a causal mask, token jj’s key and value depend only on tokens 1…j1 \dots j; nothing that comes later can change them. So once computed, they can be stored and reused. That store is the KV cache.

One decode step: the new token's query attends to every cached key and value

At each decode step, the newest token computes its own qq, kk, vv, compares qq against every cached key, mixes the cached values, and appends its own kk and vv to the cache. Without the cache, every step would recompute the whole prefix. With it, each step does a small amount of new work, but it has to read the entire cache (and all the model’s weights). That is why decode is limited by memory bandwidth, not arithmetic.

Heads: how many K/V pairs per token?

Attention runs in several heads in parallel, each looking for different patterns. The design choice that matters for serving is how many key/value heads there are:

Multi-head, grouped-query, and multi-query attention

  • Multi-head attention (MHA): every query head has its own key and value head.
  • Grouped-query attention (GQA): query heads share K/V heads in groups. Llama-3.1-70B has 64 query heads but only 8 K/V heads.
  • Multi-query attention (MQA): all query heads share a single K/V head.
  • Multi-head latent attention (MLA), used by DeepSeek models, goes further: it caches a compressed latent vector instead of full keys and values.

The KV cache per token is

bytes per token=2⋅L⋅hkv⋅dh⋅b,\text{bytes per token} = 2 \cdot L \cdot h_{kv} \cdot d_h \cdot b,

where the 2 counts one key and one value, LL is the number of layers, hkvh_{kv} the number of K/V heads, dhd_h the head dimension, and bb the bytes per value (2 in BF16). For Llama-3.1-70B in BF16, that’s 2×80×8×128×2=327,6802 \times 80 \times 8 \times 128 \times 2 = 327{,}680 bytes, or 320 KiB per token. With full multi-head attention it would be eight times larger. GQA is a big part of why serving long contexts is possible at all, and why a KV cache can be moved between GPUs in tens of milliseconds rather than seconds.

The takeaway

  1. Masks decide which models generate. Encoder-only models are all prefill; decoder-only and encoder-decoder models also decode.
  2. Causality makes caching valid. A token’s keys and values never change once computed.
  3. Decode reads everything every step: all the weights and the whole cache. That’s what makes it memory-bound.
  4. K/V heads set the cache size, and the cache size sets how expensive it is to store, and to move.