← All refreshers Refresher · 6 min read

TTFT, ITL, TPOT, percentiles, and goodput

The handful of numbers every LLM serving benchmark reports, what each one measures from a user's point of view, and why averages hide what users feel.

Skip if You know the difference between ITL and TPOT, and why a p99 matters more than a mean for streaming.

One request, four numbers

TTFT, ITL, and TPOT on one request's timeline

  • TTFT, time to first token: from the moment a request arrives until its first output token. Mostly the prefill, plus any queueing. It’s how long a user stares at a blank screen.
  • ITL, inter-token latency: the gap between two consecutive output tokens. It decides whether text streams smoothly or stutters.
  • TPOT, time per output token: the average ITL across one answer. Benchmarks often use it as the decode-speed target.
  • E2E latency: TTFT plus every ITL; the total time until the last token.

The trap is in TPOT. A single 700 ms stall in the middle of a 256-token answer raises its TPOT by less than 3 ms, a rounding error, but the user saw the answer freeze. An average per-token target can be met by an answer that visibly stuttered.

Percentiles: what most users get vs. what some users get

Latency is reported as percentiles over many requests or many tokens. The p50 (median) is the typical experience: half are faster, half slower. The p99 is the experience of the unluckiest 1 in 100. For streaming, the p99 ITL matters most: at 30 tokens per second, a user reading a 300-token answer sees about 300 gaps, so a 1-in-100 stall happens in practically every answer.

Throughput vs. goodput

  • Throughput counts everything the server produced: tokens or requests per second, regardless of how long anyone waited.
  • Goodput counts only what met its targets. A common definition: the highest request rate at which at least a given share of requests (say 90%) meet both a TTFT target and a decode-speed target.

The two can move in opposite directions. Packing more work onto the same GPUs usually raises throughput while pushing more requests past their latency targets, so goodput falls. A serving design is only better if it’s better at the goodput that matches what your users need, which is why the choice of target (average or worst gap, p50 or p99) is part of the design.