serving
FDE interview questions tagged serving, across every topic.
5 questions · 1 unlocked for you
Concepts behind "serving"
The curriculum that explains the ideas these questions test.
Core
KV CacheDuring autoregressive decoding a model would recompute attention over every prior token at each step; the KV cache stores each token's key and value vectors so each new token only attends, never recomputes, making per-token generation far cheaper. The cache grows with sequence length times batch size and becomes the memory bottleneck in serving, which is what motivates PagedAttention. FDE loops probe it because it explains why long contexts are costly to serve and why throughput, not the model, is often the constraint.🧠 Foundations of LLMs & GenAISign in
Core
Latency OptimizationMeasure p50, p95, and p99 before you touch anything, then find where the time actually goes: tokenization, retrieval, inference, or post-processing. A naive RAG pipeline that takes 1.5 seconds can usually reach sub-100ms perceived latency by caching, parallelizing retrieval, picking a smaller model, and streaming the first token, in that order of payoff.⚙️ System Design for AI in ProductionSign in
Core
Inference Serving (vLLM, TGI)Serving LLMs at high throughput under a latency SLO is its own engineering problem. Continuous batching keeps the GPU busy across requests of different lengths, and PagedAttention stops the KV cache from wasting memory. Naive one-request-at-a-time serving leaves most of an expensive GPU idle, which is why purpose-built runtimes like vLLM and TGI exist.🖥️ ML Infrastructure & ServingSign in
Advanced
Feature StoresA feature store is a central place that computes a feature once and serves it to both training (offline, batch) and serving (online, low-latency) from the same definition, which kills the most common production bug in ML: train/serve skew. It also handles point-in-time correctness so backfills do not leak the future. The honest catch is that most early-stage teams do not need one.🔁 MLOps & Lifecycle🔒 Premium
Advanced
Continuous BatchingStatic batching runs a fixed group of requests to completion together, so a batch of one short reply and one long reply makes the GPU idle while it waits on the longest. Continuous batching adds and evicts sequences from the running batch every decode step, keeping the GPU saturated and multiplying throughput. It is the scheduling trick at the heart of vLLM and every modern LLM serving stack.🖥️ ML Infrastructure & Serving🔒 Premium
Advanced
PagedAttentionAllocating each sequence's KV cache as one contiguous block forces you to reserve space for the maximum possible length, which fragments GPU memory and wastes most of it. PagedAttention borrows OS-style paging: it stores the KV cache in fixed-size non-contiguous blocks tracked by a per-sequence block table, so memory is packed, grown on demand, and shareable across sequences. It is the core memory trick that lets vLLM pack far more concurrent sequences into the same VRAM.🖥️ ML Infrastructure & Serving🔒 Premium