FDEInterviews logo
← ML Infrastructure & GPUs / 31
hardNVIDIAAnthropicxAI

A 256-GPU job hangs at a collective with every GPU at 100%. Triage the NCCL hang.

The worst failure mode in distributed training: no crash, no log, every GPU pinned at 100%, and the job dead. The triage order that finds a collective mismatch or a dead rank before you burn a day of cluster time.

Updated Sep 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.

The worst failure mode in distributed training: no crash, no log, every GPU pinned at 100%, and the job dead. The triage order that finds a collective mismatch or a dead rank before you burn a day of cluster time.

20 answers per topic instead of 10, plus saved progress and bookmarks · no cardor unlock all 523 remaining answers · ₹2,000 / $25
UP NEXT ON YOUR JOURNEY
Next in this trackServe a 400B-class MoE under a 200ms p99 inter-token SLA. How do you lay it out?Next in this trackPlan a year of GPU fleet capacity and cost for a growing inference business. Buy, reserve, or burst?Next in this trackDesign the failure domains for a 16k-GPU training cluster. What's the blast radius of a single fault?
FEDITOR'S NOTE

The trap is treating a hang like a crash and grepping for a stack trace that never comes. Strong candidates know NCCL spins as a kernel so the symptom is silence plus 100% utilization, reach for the watchdog timeout and flight recorder first, and can name the top causes (collective shape mismatch, a rank that died or diverged, fabric manager down) without flailing.

DISCUSSION · 0

No comments yet — be the first to share your approach.