Have you worked on machine learning GPU workloads?

Question

FDEInterviews · Accepted Answer

TL;DR: Answer in the order scope, scale, stack, then one quantified optimization story, and place your honest depth marker deliberately because the interviewer will drill exactly one level past wherever you stop. The story (symptom, diagnosis method, fix, number) is what they remember, and inflating scale is the fastest way to fail the pre-loaded follow-up. How to approach it This question is a fork: your answer determines which follow-ups you get, and the interviewer will drill exactly one level deeper than wherever you stop. So structure your answer as scope, scale, stack, then one optimization story with numbers, and place your honest depth marker deliberately. Never inflate. "What GPU type, and how did you profile it?" is the cheapest bluff-detector in this interview genre. A strong answer A strong response covers four things in about ninety seconds: 1. Scope: training, inference, or both, and your role. "I owned the inference serving path" reads very differently from "I submitted training jobs to a shared cluster," and both are fine; vagueness is not. 2. Scale and hardware: model sizes, GPU types and counts, single-node vs distributed. "Fine-tuned 7B-13B models on a node of 8×A100-80GB, served them on L4s" is concrete and immediately calibrates the rest of the conversation. 3. Stack: name real components. PyTorch with FSDP or DeepSpeed, vLLM or TensorRT-LLM for serving, Triton kernels if you wrote any, Slurm or Kubernetes for scheduling, W&B for tracking. Naming the orchestration and profiling tools matters as much as the framework. 4. One quantified story, the part they will remember. Pattern: symptom, diagnosis method, fix, number. For example: "Our fine-tuning throughput looked GPU-bound, but Nsight Systems showed the GPUs idle 40% of step time waiting on the dataloader; we moved tokenization offline and went from ~180 to ~310 samples/sec, which cut the job from 11 hours to 6." The diagnosis method is the detail that counts; it proves you did the work. If your experience is modest, say so and pivot to depth-per-GPU rather than GPU count: "I haven't run multi-node training, but I have done serious single-node work (memory profiling, mixed precision, batch-size tuning against OOM) and I understand the multi-node failure modes from X." Interviewers at CoreWeave and Together AI hire people who know their actual edge precisely; they reject people whose claimed edge collapses under two follow-ups. Close with one failure or war story if invited: an OOM hunt, a NaN-loss debugging session, a CUDA version mismatch that poisoned a fleet. Operating pain is the credential this question is really checking for. What interviewers probe next "What did you profile with, and what did you find?" Have a real tool (Nsight Systems/Compute, PyTorch profiler, nvidia-smi dmon) and a real finding; "we looked at GPU utilization" alone signals surface contact. "What was your GPU memory budget and what consumed it?" For training: params, gradients, optimizer states, activations; for inference: weights plus KV cache. Being able to decompose memory from memory is a strong signal, so have the standard mixed-precision-Adam arithmetic ready for whatever model size you claim (verified here for 13B): BF16 params 26 GB, BF16 gradients 26 GB, FP32 master weights 52 GB, Adam's two FP32 moments 104 GB, totaling 208 GB, the famous ~16 bytes per parameter, before a single activation. That one line answers three follow-ups at once: why the 13B run cannot fit on an 80GB card without sharding, what ZeRO/FSDP actually shard (mostly those optimizer states, the biggest slab), and why inference of the same model needs only the 26 GB of weights plus KV cache. Reciting the decomposition for your own claimed workload, unprompted, is the moment the interviewer stops probing scale and starts discussing engineering. "Why that hardware?" Cost, availability, and memory reasoning ("A100-80GB because the 13B run with Adam states did not fit on 40GB without sharding") shows you made decisions, not just used what was there. "What broke?" They want operational scar tissue; pick the story where you found root cause, not the one where you rebooted. Common mistakes Inflating scale. Claiming "distributed training experience" that turns out to be devicemap="auto" across 2 GPUs ends the interview politely. The follow-ups exist precisely to catch this. Listing frameworks with no story. A tool inventory without a symptom-to-fix-to-number narrative is indistinguishable from a resume keyword scan. No numbers anywhere: throughput, memory, cost, or time-to-train. GPU work is quantitative; an answer without a single number sounds secondhand. Forgetting the economics. Senior candidates volunteer cost awareness ("that run was about $2k of compute, so we validated on a 1B model first"), the cheapest way to sound like an owner rather than a user. Key takeaways Structure scope, scale, stack, story; the quantified story carries the answer. Mark your honest ceiling on purpose,…