Mixture of experts: routing, expert parallelism, and why attention and experts behave differently

#The FFN half of every layer

A transformer layer has two halves: attention, where tokens look at each other, and a feed-forward network (FFN), where each token is processed on its own by a large two-layer network. In a dense model, every token goes through the same FFN, and the FFN holds most of the model’s parameters.

A mixture-of-experts (MoE) layer replaces that one FFN with $E$ smaller FFNs, the experts, plus a tiny router. For each token, the router scores the experts and sends the token to its top $k$. The token’s output is the router-weighted sum of those $k$ experts’ outputs.

#Total vs. active parameters

DeepSeek-V3 has 256 routed experts per MoE layer and sends each token to 8 of them. The model has 671B parameters in total, but each token only touches about 37B. That’s the appeal: the knowledge capacity of a huge model at roughly the compute cost of a much smaller one. The catch is memory. All 671B parameters must sit in GPU memory somewhere, even though any one token uses a few percent of them.

#Expert parallelism: dispatch and combine

No single GPU holds all the experts of a large MoE, so they’re spread across many GPUs: expert parallelism (EP). Every MoE layer then needs two network exchanges:

1. Dispatch: each GPU sends each of its tokens to the GPUs that host that token’s chosen experts.
2. Combine: the expert outputs travel back and are summed.

These all-to-all exchanges happen at every MoE layer of every forward pass, so their latency sits directly on the per-token critical path. That’s why specialized communication libraries exist for this exact pattern.

#How many tokens does an expert see?

Whether an expert computes efficiently depends on how many tokens reach it in one step. With $T$ tokens in a step, top-$k$ routing, and $E$ experts, the average is

An expert’s matrix multiply reads its whole weight matrix once per step, so its arithmetic intensity is roughly the number of tokens it processes (the roofline idea from the main post). In decode, each request contributes one token per step. A batch of 32 requests per GPU across 64 GPUs gives $T = 2{,}048$, so each of 256 experts sees $2{,}048 \times 8 / 256 = 64$ tokens per step. That’s far below the few hundred needed to keep a modern GPU’s arithmetic units busy. MoE decode is memory-bound, often badly. The way out is to gather more tokens per expert: bigger batches, or more GPUs feeding the same experts.

#Why attention and experts want different hardware

• Attention is stateful. It owns each request’s KV cache, which grows with the conversation; its memory needs scale with context length and the number of active requests. It also carries the request bookkeeping: scheduling and sampling.
• Experts are stateless. They hold huge, fixed weights and need many tokens per step to run efficiently, with no per-request memory at all.

Running both on the same GPUs forces one layout and one ratio on two workloads that would each prefer something different. That tension is exactly what attention–FFN disaggregation addresses.