memory
FDE interview questions tagged memory, across every topic.
5 questions · 0 unlocked for you
Concepts behind "memory"
The curriculum that explains the ideas these questions test.
Core
GPU Memory and VRAMVRAM is the budget that decides which models you can actually run. It is spent on three things: model weights, the KV cache, and activations. Knowing the back-of-envelope arithmetic (a 7B model at fp16 is roughly 14GB of weights) is what separates a candidate who has deployed an LLM from one who has only read about it.🖥️ ML Infrastructure & ServingSign in
Core
QuantizationQuantization stores model weights (and sometimes activations) in fewer bits, fp16 down to int8 or 4-bit, which cuts memory and speeds inference. The quality hit is usually small at int8 and larger at 4-bit. Knowing post-training quantization versus quantization-aware training, and when each is acceptable, is standard FDE interview ground.🖥️ ML Infrastructure & ServingSign in
Advanced
PagedAttentionAllocating each sequence's KV cache as one contiguous block forces you to reserve space for the maximum possible length, which fragments GPU memory and wastes most of it. PagedAttention borrows OS-style paging: it stores the KV cache in fixed-size non-contiguous blocks tracked by a per-sequence block table, so memory is packed, grown on demand, and shareable across sequences. It is the core memory trick that lets vLLM pack far more concurrent sequences into the same VRAM.🖥️ ML Infrastructure & Serving🔒 Premium