15How do you handle expert load balancing in an MoE model? What happens if 90% of tokens route to 10% of experts?▼hardOpenAIxAITogether AI1 replies○ sign inThe MoE question frontier labs use to test whether you see routing as a systems problem, not just a modeling trick. Covers the collapse spiral, auxiliary losses, capacity factors, and what changes at serving time.Open full answer →
32Serve a 400B-class MoE under a 200ms p99 inter-token SLA. How do you lay it out?▼hardOpenAIDeepSeekTogether AI2 replies◆ premiumA sparse model with few active params per token, a brutal latency target, and an all-to-all on the critical path. The parallelism layout that hits the SLA, why expert parallelism not tensor parallelism is the lever, and the tail the all-to-all hides.Open full answer →
39Design GPT-scale MoE inference as a global service across regions. How do you lay it out?▼hardOpenAIGoogleDeepSeek1 replies◆ premiumA very large sparse model, hundreds of experts, and users on every continent. The layout that keeps experts close to compute, routes around regional skew, and decides what 'global KV cache' can and cannot mean.Open full answer →
