moe

The MoE question frontier labs use to test whether you see routing as a systems problem, not just a modeling trick. Covers the collapse spiral, auxiliary losses, capacity factors, and what changes at serving time.

A sparse model with few active params per token, a brutal latency target, and an all-to-all on the critical path. The parallelism layout that hits the SLA, why expert parallelism not tensor parallelism is the lever, and the tail the all-to-all hides.

A very large sparse model, hundreds of experts, and users on every continent. The layout that keeps experts close to compute, routes around regional skew, and decides what 'global KV cache' can and cannot mean.