Without a KV cache, generating token N re-attends over all N-1 prior tokens from scratch every step, turning decoding quadratic. The cache stores each layer's past keys and values so each new token costs one forward step, and its size is what caps your batch and context.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Without a KV cache, generating token N re-attends over all N-1 prior tokens from scratch every step, turning decoding quadratic. The cache stores each layer's past keys and values so each new token costs one forward step, and its size is what caps your batch and context.
The screen is whether the candidate knows the cache stores per-layer attention keys and values for all prior tokens (not the activations, not the logits), why that turns per-token cost from re-processing the whole sequence into a single new-token step, and the scaling law: memory ~ 2 * batch * seq_len * layers * kv_heads * head_dim * bytes. The reserved follow-up: 'so how do you shrink it' (GQA/MQA, quantizing the cache, paged attention). Saying it 'caches the output' or forgetting the linear growth with sequence and batch is the tell.
No comments yet — be the first to share your approach.
