10What do ZeRO and FSDP actually shard, and how much memory does each stage save? Where does gradient checkpointing fit?▼mediumMetaOpenAIAnthropic2 repliesunlockedThe 16-bytes-per-parameter breakdown that makes ZeRO's three stages obvious instead of memorized, and the communication bill each stage runs up in exchange.Open full answer →
11We need to train a 100B-parameter model that won't fit in memory. Design the data and model parallelism.▼hardGoogleAnthropicOpenAI1 replies○ sign inA reported DeepMind research-engineer question. The winning answer opens with a memory budget in bytes, not a list of parallelism buzzwords, here's the full arithmetic and the layout it forces.Open full answer →
