Continuous Batching

Static batching runs a fixed group of requests to completion together, so a batch of one short reply and one long reply makes the GPU idle while it waits on the longest. Continuous batching adds and evicts sequences from the running batch every decode step, keeping the GPU saturated and multiplying throughput. It is the scheduling trick at the heart of vLLM and every modern LLM serving stack.