Together AI ML Infrastructure & GPUs interview questions

The roofline model in one ratio: FLOPs per byte against the hardware's ridge point. Get the H100 arithmetic right and you can classify any kernel, including why LLM decode will never be compute-bound at batch 1.

Sounds like small talk; it's actually a depth probe with follow-ups pre-loaded. A scoping framework that works whether you've run 4 GPUs or 4,000, and the bluff that ends interviews.

The MoE question frontier labs use to test whether you see routing as a systems problem, not just a modeling trick. Covers the collapse spiral, auxiliary losses, capacity factors, and what changes at serving time.

One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.

The OS virtual-memory trick that made vLLM the default serving stack. The answer interviewers want quantifies the fragmentation it killed, and knows what it cost in exchange.

Why the batch is a pool of decode slots, not a bus that waits until it's full. The mental-model shift behind every modern serving stack, and the scheduling tradeoffs interviewers push on next.

One forward pass, two opposite performance regimes. The arithmetic-intensity argument that explains TTFT vs TPOT, GPU pool design, and why the big serving stacks split the phases onto different hardware.

Free tokens from idle FLOPs, losslessly, which surprises most candidates. The mechanism, the acceptance-rate math, and the high-batch regime where speculation quietly stops paying.

Bits buy bandwidth, bandwidth buys tokens, until quality quietly craters on code and math. A decision framework with the numbers, what GPTQ and AWQ actually do differently, and the eval discipline that keeps you out of trouble.

TP=2, 4, or 8, the answer is a memory budget, a latency target, and one rule: smallest TP that fits, scale with replicas. The worked arithmetic that turns this from vibes into a defensible call.

The back-of-envelope chain every inference-platform round expects: traffic → tokens/sec → per-GPU throughput from bandwidth math → fleet size → dollars. With the headroom factors candidates forget.

A sparse model with few active params per token, a brutal latency target, and an all-to-all on the critical path. The parallelism layout that hits the SLA, why expert parallelism not tensor parallelism is the lever, and the tail the all-to-all hides.

The CFO-facing infra question: a demand forecast with a fat tail, GPUs that take months to land, and three cost structures that win in different regimes. The layered commitment model that keeps you from paying for peak all year.

The fleet was fine on 8k contexts and falls over when a few 200k-token requests arrive. The math that explains why, the admission and pooling fixes that contain it, and why throwing GPUs at it is the wrong first move.

Three serving runtimes that look interchangeable on a benchmark slide and aren't. The axes that actually separate them (scheduler, kernels, operability) and the one-line rule for which to reach for.

Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.

The NVIDIA warm-up that exposes whether you've written a kernel or just called .cuda(). The mapping from software hierarchy to silicon explains every GPU performance conversation that follows.

The two access-pattern killers behind most 'my kernel is 10x slower than cuBLAS' mysteries. The matrix transpose walks straight into both, which is why NVIDIA loves asking about it.

Kernel optimization is mostly the art of feeding 100+ TFLOPs from the small fast end of this hierarchy. Know the rough latencies and bandwidths, interviewers listen for the numbers.

The backbone question of every frontier-lab infra loop. The answer that scores is organized around what each strategy communicates and how often, not just what it splits.

The one derivation every GPU-infra loop expects on a whiteboard: scatter-reduce plus all-gather, 2(N−1)K/N per GPU, and why that's provably near-optimal. Plus the latency catch that motivates tree algorithms.

Everyone says 'maximize occupancy.' The candidates who get hired at NVIDIA know when 25% occupancy beats 75%, and can explain the resource math that decides it.