DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

Generating a response has two phases. Prefill processes the whole prompt at once and produces the first token; it is compute-heavy and determines time to first token (TTFT). Decode then produces one token per step; each step is light on compute but must read the model and KV cache, so it is memory-bound, and it determines time per output token (TPOT). Applications care about these differently: a chatbot wants a fast first token and a reading-speed stream; summarization cares more about how fast the rest of the text arrives.

Systems like vLLM run both phases on the same GPUs and batch them together to maximize total throughput. Yinmin Zhong, Hao Zhang and colleagues at Peking University, StepFun and UC San Diego show this hurts once latency targets matter. A long prefill added to a batch stalls every decoding request in it, raising TPOT; decoding work slows prefill, raising TTFT. Prioritizing one phase hurts the other. And both phases must use the same GPU count and parallelism, though they would prefer different ones.

The fix is to disaggregate: run prefill on some GPUs and decode on others, and send each request's KV cache from the prefill GPU to a decode GPU after the first token. Each phase can then be sized and parallelized for its own latency target, and several prefill instances can feed one decode instance so decode runs large, efficient batches. DistServe includes a search that, given the model, workload and both targets, picks the number of GPUs, parallelism and placement for each phase, using a simulator to estimate SLO attainment. It places paired prefill and decode stages on the same machine so KV transfers use fast NVLink.

The measure is goodput: the highest request rate per GPU at which 90% of requests meet both targets. In the paper's opening example, a 13B model on one A100 reached about 1.6 requests per second colocated, while two prefill GPUs and one decode GPU together served 10, or 3.3 per GPU. Across chatbot, code completion and summarization workloads, DistServe served up to 7.4× more requests, or held 12.6× tighter latency targets, than vLLM and DeepSpeed-MII.

When a customer self-hosts a model with real latency commitments, this paper changes how to plan capacity. Throughput alone is the wrong number to optimize.

Set two latency targets, not one. Time to first token and time per output token behave differently and trade off against each other under load. Ask the customer which matters for each use case: an assistant users watch type needs fast first tokens; a background summarizer can wait for the first token but should stream quickly.

Measure goodput, not throughput. Tokens per second across all users can look great while half the requests miss their latency target. Count requests per second per GPU that meet both targets for a set share of requests (the paper uses 90%).

Different workloads want different splits. Long prompts with short answers (summarization, RAG over large documents) are prefill-heavy; short prompts with long answers (writing, chat) are decode-heavy. The right ratio of prefill to decode capacity follows from the customer's actual length distribution.

Disaggregation is now mainstream. Several production serving stacks offer separate prefill and decode pools. When a customer outgrows a single pool, this is a standard next step, especially with strict latency commitments.

It needs fast interconnects. Moving KV caches is cheap only with high bandwidth, such as NVLink within a node or a fast cluster network. On commodity networking, plan placement carefully or the transfers eat the gains.