Core
GPU Memory and VRAM
VRAM is the budget that decides which models you can actually run. It is spent on three things: model weights, the KV cache, and activations. Knowing the back-of-envelope arithmetic (a 7B model at fp16 is roughly 14GB of weights) is what separates a candidate who has deployed an LLM from one who has only read about it.
a free account unlocks the core curriculum tier · no card
Related course lessons
Related concepts
Practise this in questions
ML Infrastructure & GPUsWhat is the KV cache, why does it dominate serving memory, and how do you size it? Do the math for a 70B model. →medium · Free with sign-inML Infrastructure & GPUsA 70B model won't fit on one 80GB GPU. Give three ways to serve it and the trade-offs. →medium · PremiumLLM & GenAI FundamentalsExplain FlashAttention: why it never writes the full N x N matrix, and how it speeds up attention without changing the math. →hard · PremiumSystem Design & Production EngineeringWhen a user closes the browser mid-inference, how do you propagate cancellation and actually stop the GPU work? →hard · PremiumLLM & GenAI FundamentalsWalk me through what actually happens when an LLM generates a response. →easy · FreeML Infrastructure & GPUsHave you worked on machine learning GPU workloads? →easy · Free