One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.
more free answers with an account · no card
medium★ EssentialOpenAIAnthropicNVIDIA
What is the KV cache, why does it dominate serving memory, and how do you size it? Do the math for a 70B model.
One formula, 2 × layers × KV heads × head_dim × bytes, unlocks every LLM serving capacity question. Worked through for Llama-70B, plus the mitigation stack from GQA to paging.
Updated Aug 2026 · Grounded in real Forward Deployed Engineer interview loops and written to a senior-engineer editorial bar.
Related course lessons
Keep practising
Next in this trackHow does vLLM's PagedAttention work, and what problem does it actually solve?Free with sign-inNext in this trackExplain continuous batching vs static batching for LLM serving.PremiumNext in this trackPrefill vs decode, why are the two phases bottlenecked differently, and why disaggregate them?Premium
DISCUSSION · 0
No comments yet — be the first to share your approach.