Efficient Memory Management for Large Language Model Serving with PagedAttention

When a language model generates text, it keeps the keys and values of every previous token for every layer: the KV cache. For a 13-billion-parameter model it takes about 800 KB per token, so a single 2,048-token request can need 1.6 GB. Serving throughput depends on how many requests you can batch together, and on a 40 GB A100 serving a 13B model, about 65% of memory holds the weights and about 30% is left for KV caches. How that 30% is managed decides how many requests run at once.

Woosuk Kwon, Zhuohan Li and colleagues at Berkeley (with Ion Stoica and Joseph Gonzalez) found that existing systems wasted most of it. Deep learning frameworks want tensors in contiguous memory, so servers reserved a contiguous chunk for each request sized to its maximum possible length, since output length is unknown in advance. Most of that space sat unused: reserved for tokens not yet generated, never used because the answer was short, or lost to fragmentation from the memory allocator. Only 20.4% to 38.2% of KV memory actually held token states.

PagedAttention borrows the operating system's answer to the same problem: virtual memory with paging. Each request's KV cache is split into blocks of a fixed number of tokens (16 by default). Blocks are allocated only when needed and can sit anywhere in GPU memory; a block table maps a request's logical blocks to physical ones, and the attention kernel reads keys and values through that table. Waste is limited to the unfilled part of each request's last block. Because blocks are referenced through tables, requests can share them: parallel samples from the same prompt share the prompt's blocks, beam-search candidates share common history, and requests with a common system prompt share its cache, with copy-on-write when one needs to diverge.

vLLM, the serving engine built around it, put 96.3% of KV memory to use, sustained 1.7 to 2.7 times the request rate of an idealized version of Orca, then the leading research system, and up to 22 times FasterTransformer at the same latency.

vLLM became one of the default open-source serving engines, and the ideas here, paged KV cache, continuous batching and prefix sharing, are now standard across serving stacks. When a customer self-hosts a model, this paper explains most of what determines cost per request.

KV cache memory sets your capacity. Concurrency, and therefore throughput, is limited by how many requests' KV caches fit in GPU memory after the weights. Long prompts and long outputs consume it quickly. Capacity planning starts with KV bytes per token for the model and the customer's real length distribution.

Long contexts cost more than they look. Every token in context stays in the cache for the whole request. A customer who sends 30,000-token prompts can fit far fewer concurrent users than one who sends 2,000-token prompts.

Shared prefixes are free capacity. System prompts, few-shot examples and retrieved context reused across requests can share KV blocks. In the paper, sharing a five-example prefix gave 3.58× the throughput of the baseline. Keep shared instructions identical and at the start of the prompt so engines can reuse them.

Sampling several answers is cheaper than it seems. Parallel samples and beam search share the prompt's KV cache, saving up to two-thirds of memory in the paper's beam-search runs.

Know the knobs. vLLM exposes the share of GPU memory for KV cache, block size, maximum sequence length and preemption behavior. Setting the maximum model length to what the customer actually needs, rather than the model's maximum, can matter.