llm serving

The reported Anthropic open-ended design, and the round where candidates get tested on GPU memory and KV-cache arithmetic mid-whiteboard. The serving spine, the memory math that explains every product limit, and the safety layer the rubric explicitly wants.

A Copilot/Cursor-style assistant lives or dies on the latency budget. The strong answer spends every millisecond of a 200ms end-to-end budget on purpose, separates the inline-ghost-text path from the chat path, and shows why prefix caching and speculative decoding are the only ways to hit it at 10k concurrent users.

One GPU, callers blocking on the response, and a hardware unit that is only efficient when fed wide batches. The whole job is trading a few milliseconds of queue wait for a large jump in throughput without ever stalling a request.

The GPU-platform capstone: dozens of models, spiky traffic, a fixed monthly GPU budget, and a p99 SLO that scale-to-zero would wreck. Token-based autoscaling, the three layers of LLM cold start, and the residency tiering that makes the budget math work.

One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.

Why the batch is a pool of decode slots, not a bus that waits until it's full. The mental-model shift behind every modern serving stack, and the scheduling tradeoffs interviewers push on next.

One forward pass, two opposite performance regimes. The arithmetic-intensity argument that explains TTFT vs TPOT, GPU pool design, and why the big serving stacks split the phases onto different hardware.

Free tokens from idle FLOPs, losslessly, which surprises most candidates. The mechanism, the acceptance-rate math, and the high-batch regime where speculation quietly stops paying.

TP=2, 4, or 8, the answer is a memory budget, a latency target, and one rule: smallest TP that fits, scale with replicas. The worked arithmetic that turns this from vibes into a defensible call.

The back-of-envelope chain every inference-platform round expects: traffic → tokens/sec → per-GPU throughput from bandwidth math → fleet size → dollars. With the headroom factors candidates forget.

The OpenAI capstone design. What separates passing answers: token-based capacity math, KV/prefix-aware routing instead of round-robin, and a degradation ladder for the day demand doubles overnight.

A sparse model with few active params per token, a brutal latency target, and an all-to-all on the critical path. The parallelism layout that hits the SLA, why expert parallelism not tensor parallelism is the lever, and the tail the all-to-all hides.

Three serving runtimes that look interchangeable on a benchmark slide and aren't. The axes that actually separate them (scheduler, kernels, operability) and the one-line rule for which to reach for.

Do the weight math first, then choose your poison: shrink the weights, split them across GPUs, or spill to host memory. Each buys you the fit at a different cost, and one of them is almost always wrong for production.

A very large sparse model, hundreds of experts, and users on every continent. The layout that keeps experts close to compute, routes around regional skew, and decides what 'global KV cache' can and cannot mean.

Reprefilling the whole history every turn makes a long chat O(n^2). The real design is a radix tree of KV blocks with copy-on-write branching, a weight-epoch tag, and an eviction policy that protects live turns.

Each customer has their own LoRA on a shared 8B. The S-LoRA-style answer keeps one base in memory, pages adapters in and out, and tags every KV block with an adapter ID so reuse never crosses tenants.