Inference

Serving mixture-of-experts models

A model with a small active size is cheap per token only after its full weights fit somewhere fast. Plan for both numbers.

By LLM Model Notes · 4 November 2026 · 9 minute read

Many recent open-weight releases from Chinese labs are mixture-of-experts models with total sizes from a few hundred billion to more than two trillion parameters and active sizes from about ten to about one hundred billion. Those two numbers lead to very different hardware plans. This note explains which costs follow which number and how to measure the result instead of estimating it from a parameter count.

How routing changes the cost model

In a dense model every parameter takes part in every token. In a mixture-of-experts model, each layer holds many expert feed-forward blocks and a small router. For each token the router picks a few experts, and only those run. Attention layers and any shared experts still run for every token.

As a result, the arithmetic for one token is close to that of a dense model with the active parameter count. The weights that must be available, however, include every expert, because the next token can route anywhere. That is the central trade-off: compute per token follows the active size, while memory capacity follows the total size.

Estimate weight memory from the total

A first estimate of weight memory is the total parameter count multiplied by the bytes per parameter of the stored format. This ignores runtime overhead, activations, and the key-value cache, so it is a lower bound.

For a model such as moonshotai/Kimi-K2.6, which states one trillion total and 32 billion active parameters, even the 4-bit row exceeds the memory of a single accelerator. The deployment needs several accelerators, a multi-node setup, or offloading of experts to slower memory. Each choice changes latency in a different way, so record which one a benchmark used.

Smaller releases change the picture. Qwen/Qwen3.6-35B-A3B states 35 billion total and 3 billion active parameters. Its weights fit on one large accelerator in 8-bit form, and its per-token compute is close to that of a 3-billion-parameter dense model.

Do not forget the key-value cache

Each request keeps keys and values for every processed token in every attention layer. This cache grows with context length and with the number of concurrent requests. It does not shrink because the feed-forward part is sparse. With context windows of 256,000 tokens or more, the cache can need more memory than the weights at high concurrency.

Several recent cards describe attention variants that reduce this cost, such as linear or hybrid attention layers mixed with full attention. The effect depends on the exact layer mix and the runtime's support for it. Measure memory per request at the context lengths you need, as described in the latency and context note.

Expert placement and parallelism

When experts are spread across devices, each token's hidden state must travel to the devices that hold its selected experts and back. This all-to-all exchange adds latency that a dense model of the same active size does not have. Its cost depends on interconnect bandwidth, batch size, and how evenly the router spreads load.

• Load balance: if many tokens pick the same experts, the devices that hold them become the bottleneck. Record per-expert load when the runtime exposes it.
• Batch size: small batches leave most experts idle, so throughput per device falls. Large batches improve utilization but increase queueing delay.
• Offloading: keeping rarely used experts in host memory saves accelerator memory but makes latency depend on routing, which can vary by language and task.

Measure, then compare

Because these effects interact, a fair comparison between a mixture-of-experts model and a dense model uses the same workload, the same service objective, and the same hardware budget. Report the hardware and parallelism layout next to the numbers.

Quality also needs a check under the exact serving configuration. Quantizing a large model so that it fits can change its answers, and the change can differ between experts that see different kinds of input. Run the task-quality check from the comparison note on the quantized build you will serve.

A serving plan checklist

• Total and active parameters, and the storage format of the weights.
• Weight memory lower bound and the layout that holds it.
• Key-value cache memory per request at each target context length.
• Expert placement, interconnect, and any offloading.
• Throughput and latency at a declared service objective.
• Task quality measured on the exact build and format you serve.

Primary references

• Switch Transformers — sparse expert routing and its training and serving trade-offs.
• DeepSeekMoE — fine-grained and shared experts in a mixture-of-experts design.
• PagedAttention — key-value cache memory management in serving.