NCCL's topology tricks, GPUDirect RDMA, and the single most misleading metric in distributed training. If you've ever trusted nvidia-smi on a slow run, this question was written for you.
more free answers with an account · no card
← ML Infrastructure & GPUs / 13
mediumNVIDIACoreWeavexAI
What does NCCL actually do, and why can GPU utilization read 100% while the job is communication-bound?
NCCL's topology tricks, GPUDirect RDMA, and the single most misleading metric in distributed training. If you've ever trusted nvidia-smi on a slow run, this question was written for you.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
UP NEXT ON YOUR JOURNEY
Next in this trackInfiniBand vs RoCE for a GPU training cluster, how do you choose?Next in this trackHow do you handle expert load balancing in an MoE model? What happens if 90% of tokens route to 10% of experts?Next in this trackYour training run compiles and runs, but the loss is flat. Walk me through your debugging process.
DISCUSSION · 0
No comments yet — be the first to share your approach.