NVIDIA ML Infrastructure & GPUs interview questions

The NVIDIA warm-up that exposes whether you've written a kernel or just called .cuda(). The mapping from software hierarchy to silicon explains every GPU performance conversation that follows.

Kernel optimization is mostly the art of feeding 100+ TFLOPs from the small fast end of this hierarchy. Know the rough latencies and bandwidths, interviewers listen for the numbers.

A staple NVIDIA screen with a precise mechanical answer, and a follow-up trap about which branches actually diverge that catches most candidates.

The two access-pattern killers behind most 'my kernel is 10x slower than cuBLAS' mysteries. The matrix transpose walks straight into both, which is why NVIDIA loves asking about it.

Everyone says 'maximize occupancy.' The candidates who get hired at NVIDIA know when 25% occupancy beats 75%, and can explain the resource math that decides it.

The roofline model in one ratio: FLOPs per byte against the hardware's ridge point. Get the H100 arithmetic right and you can classify any kernel, including why LLM decode will never be compute-bound at batch 1.

Sounds like small talk; it's actually a depth probe with follow-ups pre-loaded. A scoping framework that works whether you've run 4 GPUs or 4,000, and the bluff that ends interviews.

The backbone question of every frontier-lab infra loop. The answer that scores is organized around what each strategy communicates and how often, not just what it splits.

The one derivation every GPU-infra loop expects on a whiteboard: scatter-reduce plus all-gather, 2(N−1)K/N per GPU, and why that's provably near-optimal. Plus the latency catch that motivates tree algorithms.

A real infra-loop question that candidates fumble by guessing. The correct map, TP on NVLink, PP and DP across the scale-out fabric, falls out of three traffic profiles, and the reasoning is the answer.

NCCL's topology tricks, GPUDirect RDMA, and the single most misleading metric in distributed training. If you've ever trusted nvidia-smi on a slow run, this question was written for you.

One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.

The OS virtual-memory trick that made vLLM the default serving stack. The answer interviewers want quantifies the fragmentation it killed, and knows what it cost in exchange.

One forward pass, two opposite performance regimes. The arithmetic-intensity argument that explains TTFT vs TPOT, GPU pool design, and why the big serving stacks split the phases onto different hardware.

Free tokens from idle FLOPs, losslessly, which surprises most candidates. The mechanism, the acceptance-rate math, and the high-batch regime where speculation quietly stops paying.

Bits buy bandwidth, bandwidth buys tokens, until quality quietly craters on code and math. A decision framework with the numbers, what GPTQ and AWQ actually do differently, and the eval discipline that keeps you out of trouble.

TP=2, 4, or 8, the answer is a memory budget, a latency target, and one rule: smallest TP that fits, scale with replicas. The worked arithmetic that turns this from vibes into a defensible call.

The worst failure mode in distributed training: no crash, no log, every GPU pinned at 100%, and the job dead. The triage order that finds a collective mismatch or a dead rank before you burn a day of cluster time.

A sparse model with few active params per token, a brutal latency target, and an all-to-all on the critical path. The parallelism layout that hits the SLA, why expert parallelism not tensor parallelism is the lever, and the tail the all-to-all hides.

The fleet was fine on 8k contexts and falls over when a few 200k-token requests arrive. The math that explains why, the admission and pooling fixes that contain it, and why throwing GPUs at it is the wrong first move.

Three serving runtimes that look interchangeable on a benchmark slide and aren't. The axes that actually separate them (scheduler, kernels, operability) and the one-line rule for which to reach for.

Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.

A very large sparse model, hundreds of experts, and users on every continent. The layout that keeps experts close to compute, routes around regional skew, and decides what 'global KV cache' can and cannot mean.

Constrained JSON/grammar decoding usually means a Python callback per token, which serializes the GPU. The staff answer keeps the DFA on-device and turns the mask into a single coalesced gather.

A 7B VLM at 25 FPS on an embedded GPU sounds impossible until you stop processing every frame fully. The answer is token pruning, frame-to-frame KV reuse, and a three-stage pipeline that hides latency.

Going 4K to 1M is not one trick. It is RoPE rescaling chosen by frequency band, a long-doc continued-pretrain mix, needle evals that actually catch failures, and KV math that forces ring attention.

A reported DeepMind research-engineer question. The winning answer opens with a memory budget in bytes, not a list of parallelism buzzwords, here's the full arithmetic and the layout it forces.

The 16-bytes-per-parameter breakdown that makes ZeRO's three stages obvious instead of memorized, and the communication bill each stage runs up in exchange.

Both carry RDMA at 400Gb/s; the decision is about loss behavior, tuning burden, and who operates it. A defensible recommendation with the conditions that flip it, the format this question is scored on.

The MoE question frontier labs use to test whether you see routing as a systems problem, not just a modeling trick. Covers the collapse spiral, auxiliary losses, capacity factors, and what changes at serving time.

A reported DeepMind debugging-round prompt. What's scored is your search order, not your bug list, the overfit-one-batch ladder that finds 90% of flat-loss causes in under an hour.

At 16k GPUs something fails every few hours, Meta logged 466 interruptions in 54 days training Llama 3. The checkpoint-interval math, straggler detection, and automation that turn failures into a budget line instead of an emergency.