09Explain how ring all-reduce works and derive its communication cost.▼mediumNVIDIAOpenAIMeta1 repliesunlockedThe one derivation every GPU-infra loop expects on a whiteboard: scatter-reduce plus all-gather, 2(N−1)K/N per GPU, and why that's provably near-optimal. Plus the latency catch that motivates tree algorithms.Open full answer →
13What does NCCL actually do, and why can GPU utilization read 100% while the job is communication-bound?▼mediumNVIDIACoreWeavexAI1 replies○ sign inNCCL's topology tricks, GPUDirect RDMA, and the single most misleading metric in distributed training. If you've ever trusted nvidia-smi on a slow run, this question was written for you.Open full answer →
31A 256-GPU job hangs at a collective with every GPU at 100%. Triage the NCCL hang.▼hardNVIDIAAnthropicxAI1 replies◆ premiumThe worst failure mode in distributed training: no crash, no log, every GPU pinned at 100%, and the job dead. The triage order that finds a collective mismatch or a dead rank before you burn a day of cluster time.Open full answer →
