batching

The OpenAI FDE deep-dive where you're expected to walk the full stack (network, queueing, tokenization, prefill, KV cache, batching, decode) in order, with a measurement at each hop. The triage sequence, with the numbers that tell you which layer is guilty.

Without a KV cache, generating token N re-attends over all N-1 prior tokens from scratch every step, turning decoding quadratic. The cache stores each layer's past keys and values so each new token costs one forward step, and its size is what caps your batch and context.

Batching inputs is how you keep a GPU fed, but the source is an iterator you cannot index or len(). The warm-up that screens whether you can write a lazy generator that handles the ragged final batch.

Latency, cost, and quality are one budget with three claims on it. The strong answer treats the p99 SLA and the spend ceiling as a joint constraint, finds where they fight (batching), and names the lever it pulls when traffic exceeds what the ceiling can buy at SLA.

One GPU, callers blocking on the response, and a hardware unit that is only efficient when fed wide batches. The whole job is trading a few milliseconds of queue wait for a large jump in throughput without ever stalling a request.