Mixture of Experts (MoE)

An MoE replaces the dense feed-forward block of a transformer with many parallel expert blocks plus a small router that activates only a few experts per token. This decouples total parameter count from per-token compute: the model can hold hundreds of billions of parameters while doing the work of a much smaller one on any given token. FDE loops probe it because the headline 'huge but cheap' hides a brutal serving cost, every expert must sit in memory even though most stay idle.