← All refreshers Refresher · 7 min read

How a multimodal model reads an image

Patches, a vision transformer, a projector, and visual tokens: how an image becomes part of a language model's prompt, and what that costs.

Skip if You can say how many tokens a 640x640 image becomes and where the vision encoder sits relative to prefill.

Three stages before the language model sees anything

A large multimodal model (LMM, or vision-language model) such as Qwen3-VL is a language model with an image front-end bolted on:

  1. Patchify. The image is cut into small square patches, typically 14 or 16 pixels per side. Each patch is flattened into a vector.
  2. Encode. A vision transformer (ViT) runs attention over all the patches at once. It’s an encoder: bidirectional, every patch sees every other patch (see the attention refresher), and there’s nothing to generate, so it’s one parallel pass.
  3. Project. Neighboring patch outputs are usually merged (often 2×2), and a small projector maps each merged vector to the language model’s hidden size. Each result is a visual token: it sits in the prompt exactly where an ordinary text token’s embedding would.

From image to patches to visual tokens

Counting visual tokens

With patch size pp and an m×mm \times m merge, an image of W×HW \times H pixels becomes

nvisual=Wp⋅m×Hp⋅mn_{\text{visual}} = \frac{W}{p \cdot m} \times \frac{H}{p \cdot m}

tokens. With 16-pixel patches merged 2×2, every 32×32-pixel area becomes one token, so a 640×640 image becomes 20×20=40020 \times 20 = 400 visual tokens. Four images add 1,600 tokens to the prompt before a single word of text. Higher resolution grows the count quadratically; video multiplies it by the number of frames.

Where the cost lands

The vision encoder runs before the language model’s prefill can start, because prefill needs the visual tokens as input. So:

  • The encoder adds directly to time to first token. The heavier the images, the longer the user waits.
  • It’s compute-bound, like prefill: one parallel pass over many patches.
  • Its output is small. It’s one vector per visual token, in the language model’s hidden size: about 7 KiB per token for a model with a 3,584-wide hidden state in BF16. Compare that with the KV cache those same tokens create once prefill processes them, which is every layer’s keys and values: several times larger.

vLLM keeps an encoder cache of these outputs, keyed by a hash of the image. A repeated image (a logo, a product photo, a diagram that appears in many prompts) can skip the encoder entirely.

The takeaway

  1. Images become tokens in the prompt: hundreds per image, set by resolution, patch size, and merging.
  2. The vision encoder is a separate model stage that runs before prefill and lands on TTFT.
  3. What it hands to the language model, one embedding per visual token, is small compared with the KV cache prefill builds from it.