xAI ML Infrastructure & GPUs interview questions

Kernel optimization is mostly the art of feeding 100+ TFLOPs from the small fast end of this hierarchy. Know the rough latencies and bandwidths, interviewers listen for the numbers.

A staple NVIDIA screen with a precise mechanical answer, and a follow-up trap about which branches actually diverge that catches most candidates.

The two access-pattern killers behind most 'my kernel is 10x slower than cuBLAS' mysteries. The matrix transpose walks straight into both, which is why NVIDIA loves asking about it.

A real infra-loop question that candidates fumble by guessing. The correct map, TP on NVLink, PP and DP across the scale-out fabric, falls out of three traffic profiles, and the reasoning is the answer.

NCCL's topology tricks, GPUDirect RDMA, and the single most misleading metric in distributed training. If you've ever trusted nvidia-smi on a slow run, this question was written for you.

Both carry RDMA at 400Gb/s; the decision is about loss behavior, tuning burden, and who operates it. A defensible recommendation with the conditions that flip it, the format this question is scored on.

The MoE question frontier labs use to test whether you see routing as a systems problem, not just a modeling trick. Covers the collapse spiral, auxiliary losses, capacity factors, and what changes at serving time.

At 16k GPUs something fails every few hours, Meta logged 466 interruptions in 54 days training Llama 3. The checkpoint-interval math, straggler detection, and automation that turn failures into a budget line instead of an emergency.

The resume-walk question for infra roles, with a hidden rubric: layer ownership, scale fluency, and one defensible war story. How to answer credibly at any actual scale, and the inflation tells interviewers catch.

A reported OpenAI design prompt. The skeleton: gang scheduling, topology-aware placement, preemption with checkpoint contracts, and quota, plus the fragmentation problem that quietly eats 20% of naive clusters.

The worst failure mode in distributed training: no crash, no log, every GPU pinned at 100%, and the job dead. The triage order that finds a collective mismatch or a dead rank before you burn a day of cluster time.

One synchronous job, 16k GPUs, and any single component can stall all of them. The blast-radius analysis interviewers want: which faults take down a rack vs the run, where the shared single points of failure hide, and how parallelism layout maps onto failure domains.

No error, no hang, the loss still moves, but the run is suddenly half as fast and burning the same dollars. The triage that separates a straggler from a fabric problem from broken comm/compute overlap, using the signals nvidia-smi can't give you.

The NVIDIA warm-up that exposes whether you've written a kernel or just called .cuda(). The mapping from software hierarchy to silicon explains every GPU performance conversation that follows.

A reported DeepMind research-engineer question. The winning answer opens with a memory budget in bytes, not a list of parallelism buzzwords, here's the full arithmetic and the layout it forces.

Everyone says 'maximize occupancy.' The candidates who get hired at NVIDIA know when 25% occupancy beats 75%, and can explain the resource math that decides it.

The backbone question of every frontier-lab infra loop. The answer that scores is organized around what each strategy communicates and how often, not just what it splits.

The one derivation every GPU-infra loop expects on a whiteboard: scatter-reduce plus all-gather, 2(N−1)K/N per GPU, and why that's provably near-optimal. Plus the latency catch that motivates tree algorithms.

The roofline model in one ratio: FLOPs per byte against the hardware's ridge point. Get the H100 arithmetic right and you can classify any kernel, including why LLM decode will never be compute-bound at batch 1.