Distributed Training (FSDP, Parallelism)

When a model or its training state will not fit on one GPU, you split the work across many. Data parallelism replicates the model and splits the batch; tensor parallelism splits a single layer's math across GPUs; pipeline parallelism splits the layer stack into stages; and FSDP/ZeRO shard the parameters, gradients, and optimizer states themselves. Each buys memory by spending network bandwidth, so the real skill is composing them to fit the model while keeping the GPUs busy.