kv cache

The OpenAI FDE deep-dive where you're expected to walk the full stack (network, queueing, tokenization, prefill, KV cache, batching, decode) in order, with a measurement at each hop. The triage sequence, with the numbers that tell you which layer is guilty.

Each chat turn resends the whole conversation, so the model re-prefills the same prefix every time. Server-side KV caching and client-side cache breakpoints kill that redundant compute, but exact-prefix matching and stale entries are where teams silently get it wrong.

Without a KV cache, generating token N re-attends over all N-1 prior tokens from scratch every step, turning decoding quadratic. The cache stores each layer's past keys and values so each new token costs one forward step, and its size is what caps your batch and context.

The rate limiter is working correctly and that is the problem. It counts requests, and one hundred-thousand-token request occupies the KV cache of fifty short ones, so admitting both tenants equally hands one of them eighty percent of the machine.

The reported Anthropic open-ended design, and the round where candidates get tested on GPU memory and KV-cache arithmetic mid-whiteboard. The serving spine, the memory math that explains every product limit, and the safety layer the rubric explicitly wants.

One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.

The OS virtual-memory trick that made vLLM the default serving stack. The answer interviewers want quantifies the fragmentation it killed, and knows what it cost in exchange.

The fleet was fine on 8k contexts and falls over when a few 200k-token requests arrive. The math that explains why, the admission and pooling fixes that contain it, and why throwing GPUs at it is the wrong first move.

A very large sparse model, hundreds of experts, and users on every continent. The layout that keeps experts close to compute, routes around regional skew, and decides what 'global KV cache' can and cannot mean.

Reprefilling the whole history every turn makes a long chat O(n^2). The real design is a radix tree of KV blocks with copy-on-write branching, a weight-epoch tag, and an eviction policy that protects live turns.

Each customer has their own LoRA on a shared 8B. The S-LoRA-style answer keeps one base in memory, pages adapters in and out, and tags every KV block with an adapter ID so reuse never crosses tenants.

Going 4K to 1M is not one trick. It is RoPE rescaling chosen by frequency band, a long-doc continued-pretrain mix, needle evals that actually catch failures, and KV math that forces ring attention.