Core
Inference Serving (vLLM, TGI)
Serving LLMs at high throughput under a latency SLO is its own engineering problem. Continuous batching keeps the GPU busy across requests of different lengths, and PagedAttention stops the KV cache from wasting memory. Naive one-request-at-a-time serving leaves most of an expensive GPU idle, which is why purpose-built runtimes like vLLM and TGI exist.
a free account unlocks the core curriculum tier · no card
Related concepts
Practise this in questions
ML Infrastructure & GPUsHow does vLLM's PagedAttention work, and what problem does it actually solve? →medium · Free with sign-inML Infrastructure & GPUsExplain continuous batching vs static batching for LLM serving. →easy · PremiumLLM & GenAI FundamentalsWalk me through how you'd diagnose high latency in an LLM inference pipeline →hard · PremiumML Infrastructure & GPUsvLLM vs TensorRT-LLM vs TGI: how do they differ and when do you pick each? →medium · PremiumLLM & GenAI FundamentalsKV-caching in autoregressive decoding: what does it store, why does it cut latency, and how does its memory cost scale? →medium · PremiumSystem Design & Production EngineeringDesign inference batching for a single GPU serving up to 100 inputs per batch while users wait synchronously. Maximize utilization. →hard · Premium