throughput
FDE interview questions tagged throughput, across every topic.
4 questions · 0 unlocked for you
Concepts behind "throughput"
The curriculum that explains the ideas these questions test.
Core
Inference Serving (vLLM, TGI)Serving LLMs at high throughput under a latency SLO is its own engineering problem. Continuous batching keeps the GPU busy across requests of different lengths, and PagedAttention stops the KV cache from wasting memory. Naive one-request-at-a-time serving leaves most of an expensive GPU idle, which is why purpose-built runtimes like vLLM and TGI exist.🖥️ ML Infrastructure & ServingSign in
Advanced
Continuous BatchingStatic batching runs a fixed group of requests to completion together, so a batch of one short reply and one long reply makes the GPU idle while it waits on the longest. Continuous batching adds and evicts sequences from the running batch every decode step, keeping the GPU saturated and multiplying throughput. It is the scheduling trick at the heart of vLLM and every modern LLM serving stack.🖥️ ML Infrastructure & Serving🔒 Premium