What is the KV cache, why does it dominate serving memory, and how do you size it? Do the math for a 70B model.

One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.

Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.