Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Every request has two phases. Prefill processes the whole prompt at once; it is compute-bound and saturates the GPU even for a single request. Decode produces one token per request per iteration; it is memory-bound, since each iteration reads all the weights to do very little arithmetic, so batching many decodes together is nearly free.

Servers like vLLM and Orca batch at the iteration level and admit new requests eagerly, running their prefills first because that grows the next decode batch. The cost is a generation stall: while a multi-thousand-token prompt is processed, every ongoing response pauses, sometimes for seconds. Time between tokens (TBT) at the tail goes up with load. Older request-level batching avoids stalls but leaves the GPU underused. So operators trade throughput against latency.

Sarathi-Serve removes the trade with two pieces. Chunked prefill splits a long prompt into chunks of a few hundred tokens, computed over several iterations. Stall-free batching builds each iteration in a fixed order: all running decodes, then any partly done prefill, then new requests, until a token budget is reached. Because decode iterations have idle compute, adding a few hundred prefill tokens barely lengthens them, and no running response ever waits for a whole prompt. Iterations also become uniform, which removes pipeline bubbles when a model is split across machines.

The token budget sets the trade: 512 tokens for strict latency targets, 2,048 for relaxed ones. Built on vLLM, Sarathi-Serve served 2.6x more load than vLLM for Mistral-7B and 3.7x for Yi-34B under strict P99 TBT targets, and made pipeline parallelism over ordinary Ethernet practical for Falcon-180B.

Users notice two latencies: time to first token and how smoothly the rest streams. When a customer says "the answer freezes mid-sentence when it's busy," the cause is often a scheduler stalling decodes to run someone else's long prompt.

Ask how the serving engine admits prefills. Chunked prefill is now a standard option in major engines; whether it is on, and with what token budget, changes tail latency at load more than most other settings.

Batch size limits don't fix it. The paper found vLLM's capacity under strict latency targets barely moved across maximum batch sizes of 32, 64 and 128, because stalls come from prefill length, not batch size.

Long prompts make it worse. RAG, document and agent workloads with prompts in the thousands of tokens are where prefill interference is largest; the arXiv-summarization workload (median prompt 7,059 tokens) had the lowest capacities in the paper.