Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Mooncake is the serving platform behind Kimi, Moonshot AI's chat service, which is known for long contexts. The paper's argument: once prompts average thousands of tokens, the KV cache is the main object a serving system moves, stores and schedules around, so design the cluster around it.

Four design choices follow. Prefill and decode run on separate machines (as in DistServe and Splitwise), because prefill is compute-bound and decode is memory-bound, and a long prefill on a decode node stalls every user's next token. A distributed KV cache pool uses the GPU servers' otherwise idle CPU memory, DRAM and SSD, with blocks moved by RDMA, so a prefix computed once on any node can be reused anywhere. A global scheduler, Conductor, picks the prefill node for each request by estimating time to first token (TTFT) on each candidate: how much of the prompt it already has cached, its queue, and the time to fetch missing blocks from another node. Hot blocks get copied to more nodes as a side effect. Admission control under overload rejects a request before its prefill if the decode pool won't have room when the prefill finishes, using a prediction of decode load rather than its current value.

Results come from a model with LLaMA2-70B's architecture (not Kimi's model) on A800 nodes. Scheduling by cache cut mean TTFT from 92 s (random) or 60 s (least-loaded) to 6.3 s on 23,000 real requests. Against vLLM with the same instance count, Mooncake served 20% to 40% more on public datasets, 50% to 525% more on simulated long prompts, and about 75% more on replayed real traffic. The authors also released a one-hour, 23,608-request trace with hashed block IDs, the first public data for studying cross-request KV reuse.

Customer deployments of long-context assistants (document Q&A, codebase chat, agents with big system prompts) look like Kimi's traffic: inputs tens of times longer than outputs, and the same prefixes arriving again and again. That is where Mooncake's lessons apply.

Prefix reuse is the cheapest throughput you will find. Every token of a repeated prefix served from cache is prefill you don't compute. The trace shows the ceiling is workload-specific (55% here, up to 90% for their paper-chat service), so measure it on the customer's logs before promising a number.

Routing by load alone throws the cache away. Sending a request to the least-busy node instead of the one holding its prefix gave a mean TTFT ten times worse in the paper's test. A router that knows where prefixes live is worth more than a smarter load metric.

Plan for overload, not just for peak. GPU supply is fixed in most deployments. Deciding which requests to refuse, and refusing them before spending prefill on them, decides how much useful work the cluster does at peak.