Core
KV Cache
During autoregressive decoding a model would recompute attention over every prior token at each step; the KV cache stores each token's key and value vectors so each new token only attends, never recomputes, making per-token generation far cheaper. The cache grows with sequence length times batch size and becomes the memory bottleneck in serving, which is what motivates PagedAttention. FDE loops probe it because it explains why long contexts are costly to serve and why throughput, not the model, is often the constraint.
a free account unlocks the core curriculum tier · no card
Related course lessons
- Budget weights, resident cache and runtime memory in consistent units →
30 min · Free
Hard Deployments · syllabus - Measure throughput without losing the requests that failed →
180 min · Premium
Hard Deployments · syllabus - Test the model server's cache, adapter and shared-failure boundaries →
30 min · Premium
Hard Deployments · syllabus
Related concepts
Practise this in questions
LLM & GenAI FundamentalsWalk me through how you'd diagnose high latency in an LLM inference pipeline →hard · PremiumLLM & GenAI FundamentalsPrefix caching in a multi-turn chatbot: how do client-side and server-side caching cut redundant compute, and what breaks correctness? →medium · PremiumLLM & GenAI FundamentalsKV-caching in autoregressive decoding: what does it store, why does it cut latency, and how does its memory cost scale? →medium · PremiumRAG & Agent System DesignOne tenant started sending 100,000-token documents and everyone else's latency doubled. The rate limiter says nobody is over quota. →hard · PremiumLLM & GenAI FundamentalsA customer says your LLM app is too slow. Give me five levers to reduce latency, and their tradeoffs. →medium · Free with sign-inML Infrastructure & GPUsWhat is the KV cache, why does it dominate serving memory, and how do you size it? Do the math for a 70B model. →medium · Free with sign-in