Tensor, pipeline, data, and expert parallelism in one page

#Four ways to use more than one GPU

• Tensor parallelism (TP) splits every layer’s matrices across GPUs. Each GPU holds a slice of every weight matrix and computes its slice of every matrix multiply, then the GPUs combine partial results with an all-reduce, typically twice per layer. TP makes one request faster and lets a model larger than one GPU fit. The price is constant communication, so it wants very fast links (NVLink) and usually stays inside one server.
• Pipeline parallelism (PP) splits the model by layers: GPU 1 runs layers 1–20, GPU 2 runs layers 21–40, and so on. It communicates less than TP (only activations between stages), but a single request has to pass through the stages one after another.
• Data parallelism (DP) runs independent full copies of the model, called replicas, and spreads requests across them. It adds capacity, not speed for any single request, and needs almost no communication.
• Expert parallelism (EP) spreads a mixture-of-experts model’s experts across GPUs and routes each token to the GPUs holding its chosen experts (see the MoE refresher).

These combine. “TP=4, DP=2” on eight GPUs means two replicas, each split four ways.

#Why the phases want different choices

Prefill is compute-bound, so splitting its matrix multiplies across more GPUs cuts the time to first token almost proportionally. The all-reduces carry large messages (one hidden vector per prompt token) but usually overlap well with the heavy computation.

Decode is memory-bound, and its sizing question is different: how many streams can run at once while every step stays under the inter-token latency target? That’s limited by KV-cache memory (whatever’s left after the weights) and by the per-step time. Decode’s all-reduces are tiny, so their fixed latency, not their bandwidth, is what costs.

A shared deployment has to use one layout for both phases. Separate pools can choose independently.