inference

The warm-up that quietly filters out candidates who only know LLMs as a magic API. Nail the two-phase inference picture and you set up every later answer about latency, cost, and hallucinations.

TTFT vs tokens-per-second, the output-length lever everyone forgets, and why streaming is the highest-ROI fix that changes no latency at all. The five-lever answer with real numbers.

The OpenAI FDE deep-dive where you're expected to walk the full stack (network, queueing, tokenization, prefill, KV cache, batching, decode) in order, with a measurement at each hop. The triage sequence, with the numbers that tell you which layer is guilty.

Two compression levers with very different bills. The arithmetic that decides between them, when to stack both, and the eval discipline that keeps a 'cheaper' model from quietly tanking your hardest 10% of queries.

Attention is bottlenecked by moving the N x N score matrix to and from HBM, not by FLOPs. FlashAttention tiles the computation in fast on-chip SRAM and fuses softmax with an online running statistic, so it computes the exact same output while never materializing the full matrix.

Each chat turn resends the whole conversation, so the model re-prefills the same prefix every time. Server-side KV caching and client-side cache breakpoints kill that redundant compute, but exact-prefix matching and stale entries are where teams silently get it wrong.

Without a KV cache, generating token N re-attends over all N-1 prior tokens from scratch every step, turning decoding quadratic. The cache stores each layer's past keys and values so each new token costs one forward step, and its size is what caps your batch and context.

The reported Anthropic open-ended design, and the round where candidates get tested on GPU memory and KV-cache arithmetic mid-whiteboard. The serving spine, the memory math that explains every product limit, and the safety layer the rubric explicitly wants.

A 2-second inference and a user who closed the tab. Naive stacks keep the GPU burning to completion. Real cancellation means propagating disconnect all the way to the batch scheduler.

A real customer-interview question that rewards scar tissue: the 29-second timeout wall, retry storms that triple your own load, and connection pools sized for web traffic meeting 30-second inferences. The issues worth claiming and the fixes that prove you were there.

One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.

The OS virtual-memory trick that made vLLM the default serving stack. The answer interviewers want quantifies the fragmentation it killed, and knows what it cost in exchange.

Why the batch is a pool of decode slots, not a bus that waits until it's full. The mental-model shift behind every modern serving stack, and the scheduling tradeoffs interviewers push on next.

One forward pass, two opposite performance regimes. The arithmetic-intensity argument that explains TTFT vs TPOT, GPU pool design, and why the big serving stacks split the phases onto different hardware.

Free tokens from idle FLOPs, losslessly, which surprises most candidates. The mechanism, the acceptance-rate math, and the high-batch regime where speculation quietly stops paying.

Bits buy bandwidth, bandwidth buys tokens, until quality quietly craters on code and math. A decision framework with the numbers, what GPTQ and AWQ actually do differently, and the eval discipline that keeps you out of trouble.

TP=2, 4, or 8, the answer is a memory budget, a latency target, and one rule: smallest TP that fits, scale with replicas. The worked arithmetic that turns this from vibes into a defensible call.

The back-of-envelope chain every inference-platform round expects: traffic → tokens/sec → per-GPU throughput from bandwidth math → fleet size → dollars. With the headroom factors candidates forget.

A sparse model with few active params per token, a brutal latency target, and an all-to-all on the critical path. The parallelism layout that hits the SLA, why expert parallelism not tensor parallelism is the lever, and the tail the all-to-all hides.

The fleet was fine on 8k contexts and falls over when a few 200k-token requests arrive. The math that explains why, the admission and pooling fixes that contain it, and why throwing GPUs at it is the wrong first move.

Three serving runtimes that look interchangeable on a benchmark slide and aren't. The axes that actually separate them (scheduler, kernels, operability) and the one-line rule for which to reach for.

Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.

A very large sparse model, hundreds of experts, and users on every continent. The layout that keeps experts close to compute, routes around regional skew, and decides what 'global KV cache' can and cannot mean.