ML Infrastructure & GPU Interview Questions

The NVIDIA warm-up that exposes whether you've written a kernel or just called .cuda(). The mapping from software hierarchy to silicon explains every GPU performance conversation that follows.

Kernel optimization is mostly the art of feeding 100+ TFLOPs from the small fast end of this hierarchy. Know the rough latencies and bandwidths, interviewers listen for the numbers.

A staple NVIDIA screen with a precise mechanical answer, and a follow-up trap about which branches actually diverge that catches most candidates.

The two access-pattern killers behind most 'my kernel is 10x slower than cuBLAS' mysteries. The matrix transpose walks straight into both, which is why NVIDIA loves asking about it.

Everyone says 'maximize occupancy.' The candidates who get hired at NVIDIA know when 25% occupancy beats 75%, and can explain the resource math that decides it.

The roofline model in one ratio: FLOPs per byte against the hardware's ridge point. Get the H100 arithmetic right and you can classify any kernel, including why LLM decode will never be compute-bound at batch 1.

Sounds like small talk; it's actually a depth probe with follow-ups pre-loaded. A scoping framework that works whether you've run 4 GPUs or 4,000, and the bluff that ends interviews.

The backbone question of every frontier-lab infra loop. The answer that scores is organized around what each strategy communicates and how often, not just what it splits.

The one derivation every GPU-infra loop expects on a whiteboard: scatter-reduce plus all-gather, 2(N−1)K/N per GPU, and why that's provably near-optimal. Plus the latency catch that motivates tree algorithms.

The 16-bytes-per-parameter breakdown that makes ZeRO's three stages obvious instead of memorized, and the communication bill each stage runs up in exchange.

A reported DeepMind research-engineer question. The winning answer opens with a memory budget in bytes, not a list of parallelism buzzwords, here's the full arithmetic and the layout it forces.

A real infra-loop question that candidates fumble by guessing. The correct map, TP on NVLink, PP and DP across the scale-out fabric, falls out of three traffic profiles, and the reasoning is the answer.

NCCL's topology tricks, GPUDirect RDMA, and the single most misleading metric in distributed training. If you've ever trusted nvidia-smi on a slow run, this question was written for you.

Both carry RDMA at 400Gb/s; the decision is about loss behavior, tuning burden, and who operates it. A defensible recommendation with the conditions that flip it, the format this question is scored on.

The MoE question frontier labs use to test whether you see routing as a systems problem, not just a modeling trick. Covers the collapse spiral, auxiliary losses, capacity factors, and what changes at serving time.

A reported DeepMind debugging-round prompt. What's scored is your search order, not your bug list, the overfit-one-batch ladder that finds 90% of flat-loss causes in under an hour.

At 16k GPUs something fails every few hours, Meta logged 466 interruptions in 54 days training Llama 3. The checkpoint-interval math, straggler detection, and automation that turn failures into a budget line instead of an emergency.

The resume-walk question for infra roles, with a hidden rubric: layer ownership, scale fluency, and one defensible war story. How to answer credibly at any actual scale, and the inflation tells interviewers catch.

One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.

The OS virtual-memory trick that made vLLM the default serving stack. The answer interviewers want quantifies the fragmentation it killed, and knows what it cost in exchange.

Why the batch is a pool of decode slots, not a bus that waits until it's full. The mental-model shift behind every modern serving stack, and the scheduling tradeoffs interviewers push on next.

One forward pass, two opposite performance regimes. The arithmetic-intensity argument that explains TTFT vs TPOT, GPU pool design, and why the big serving stacks split the phases onto different hardware.

Free tokens from idle FLOPs, losslessly, which surprises most candidates. The mechanism, the acceptance-rate math, and the high-batch regime where speculation quietly stops paying.

Bits buy bandwidth, bandwidth buys tokens, until quality quietly craters on code and math. A decision framework with the numbers, what GPTQ and AWQ actually do differently, and the eval discipline that keeps you out of trouble.

TP=2, 4, or 8, the answer is a memory budget, a latency target, and one rule: smallest TP that fits, scale with replicas. The worked arithmetic that turns this from vibes into a defensible call.

torch_xla makes the demo run in an afternoon; making it fast is where migrations die. The lazy-tensor mental model, the recompilation traps, and the honest fork between porting and rewriting in JAX.

A reported OpenAI design prompt. The skeleton: gang scheduling, topology-aware placement, preemption with checkpoint contracts, and quota, plus the fragmentation problem that quietly eats 20% of naive clusters.

The back-of-envelope chain every inference-platform round expects: traffic → tokens/sec → per-GPU throughput from bandwidth math → fleet size → dollars. With the headroom factors candidates forget.

A reported OpenAI phone-screen exercise that looks trivial and is graded on edge cases: money-safe arithmetic, billing increments, unknown SKUs, and the clarifying questions you ask before typing.

The OpenAI capstone design. What separates passing answers: token-based capacity math, KV/prefix-aware routing instead of round-robin, and a degradation ladder for the day demand doubles overnight.

The worst failure mode in distributed training: no crash, no log, every GPU pinned at 100%, and the job dead. The triage order that finds a collective mismatch or a dead rank before you burn a day of cluster time.

A sparse model with few active params per token, a brutal latency target, and an all-to-all on the critical path. The parallelism layout that hits the SLA, why expert parallelism not tensor parallelism is the lever, and the tail the all-to-all hides.

The CFO-facing infra question: a demand forecast with a fat tail, GPUs that take months to land, and three cost structures that win in different regimes. The layered commitment model that keeps you from paying for peak all year.

One synchronous job, 16k GPUs, and any single component can stall all of them. The blast-radius analysis interviewers want: which faults take down a rack vs the run, where the shared single points of failure hide, and how parallelism layout maps onto failure domains.

The fleet was fine on 8k contexts and falls over when a few 200k-token requests arrive. The math that explains why, the admission and pooling fixes that contain it, and why throwing GPUs at it is the wrong first move.

No error, no hang, the loss still moves, but the run is suddenly half as fast and burning the same dollars. The triage that separates a straggler from a fabric problem from broken comm/compute overlap, using the signals nvidia-smi can't give you.

Three serving runtimes that look interchangeable on a benchmark slide and aren't. The axes that actually separate them (scheduler, kernels, operability) and the one-line rule for which to reach for.

Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.

A very large sparse model, hundreds of experts, and users on every continent. The layout that keeps experts close to compute, routes around regional skew, and decides what 'global KV cache' can and cannot mean.

Constrained JSON/grammar decoding usually means a Python callback per token, which serializes the GPU. The staff answer keeps the DFA on-device and turns the mask into a single coalesced gather.

Reprefilling the whole history every turn makes a long chat O(n^2). The real design is a radix tree of KV blocks with copy-on-write branching, a weight-epoch tag, and an eviction policy that protects live turns.

Each customer has their own LoRA on a shared 8B. The S-LoRA-style answer keeps one base in memory, pages adapters in and out, and tags every KV block with an adapter ID so reuse never crosses tenants.

A 7B VLM at 25 FPS on an embedded GPU sounds impossible until you stop processing every frame fully. The answer is token pruning, frame-to-frame KV reuse, and a three-stage pipeline that hides latency.

Going 4K to 1M is not one trick. It is RoPE rescaling chosen by frequency band, a long-doc continued-pretrain mix, needle evals that actually catch failures, and KV math that forces ring attention.

64 H100s for 30 days is a real budget with a real token count. The answer sizes the run from Chinchilla, picks parallelism for this exact cluster, and has a concrete plan for loss spikes and silent data corruption.

10 billion vectors is where raw float storage (30TB) stops being an option. The answer does the HNSW vs IVF-PQ RAM math cold and shows why PQ is the only thing that fits in memory.