scaling
FDE interview questions tagged scaling, across every topic.
4 questions · 0 unlocked for you
Concepts behind "scaling"
The curriculum that explains the ideas these questions test.
Advanced
Mixture of Experts (MoE)An MoE replaces the dense feed-forward block of a transformer with many parallel expert blocks plus a small router that activates only a few experts per token. This decouples total parameter count from per-token compute: the model can hold hundreds of billions of parameters while doing the work of a much smaller one on any given token. FDE loops probe it because the headline 'huge but cheap' hides a brutal serving cost, every expert must sit in memory even though most stay idle.🧠 Foundations of LLMs & GenAI🔒 Premium
Advanced
Distributed Training (FSDP, Parallelism)When a model or its training state will not fit on one GPU, you split the work across many. Data parallelism replicates the model and splits the batch; tensor parallelism splits a single layer's math across GPUs; pipeline parallelism splits the layer stack into stages; and FSDP/ZeRO shard the parameters, gradients, and optimizer states themselves. Each buys memory by spending network bandwidth, so the real skill is composing them to fit the model while keeping the GPUs busy.🖥️ ML Infrastructure & Serving🔒 Premium