← 🖥️ ML Infrastructure & Serving
Advanced
Distributed Training (FSDP, Parallelism)
When a model or its training state will not fit on one GPU, you split the work across many. Data parallelism replicates the model and splits the batch; tensor parallelism splits a single layer's math across GPUs; pipeline parallelism splits the layer stack into stages; and FSDP/ZeRO shard the parameters, gradients, and optimizer states themselves. Each buys memory by spending network bandwidth, so the real skill is composing them to fit the model while keeping the GPUs busy.
Unlock the full curriculum — ₹2,000 / $25every concept + every answer · 6 months · no auto-renew
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
ML Infrastructure & GPUsCompare data, tensor, and pipeline parallelism, when do you use each, and how do they combine into 3D parallelism?→ML Infrastructure & GPUsExplain how ring all-reduce works and derive its communication cost.→ML Infrastructure & GPUsWhat do ZeRO and FSDP actually shard, and how much memory does each stage save? Where does gradient checkpointing fit?→ML Infrastructure & GPUsWe need to train a 100B-parameter model that won't fit in memory. Design the data and model parallelism.→ML Infrastructure & GPUsPipeline parallelism, does it run over the scale-out network or stay inside the node? Map TP, PP, and DP onto the fabric.→ML Infrastructure & GPUsWhat does NCCL actually do, and why can GPU utilization read 100% while the job is communication-bound?→