LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference

The KV cache started as a per-request scratchpad: computed during prefill, read during decoding, discarded at the end. Two trends changed that. Prefix caching reuses one query's KV cache for later queries that share its beginning, such as a long document, a system prompt or a conversation history. Prefill–decode disaggregation computes the cache on one GPU and decodes on another. Both need KV caches to live outside a single request and often outside GPU memory, where there is too little room to keep everything worth reusing.

LMCache, from Junchen Jiang's group at the University of Chicago and the company Tensormesh, is an open-source layer that does this. It plugs into vLLM and SGLang through a connector interface, extracts KV caches as requests run, stores them across a hierarchy (GPU memory, CPU memory, local disk, remote storage such as Redis or S3), and loads them back when a later query shares the prefix. It also ships caches between prefill and decode instances.

Most of the paper is about making the data movement fast. Engines store KV in pages of 16 tokens per layer, 20 to 63 KB each, far too small to saturate a network or a PCIe link. LMCache gathers pages into 256-token chunks through a GPU staging buffer, moves them with DMA, loads the next layer's cache while the current layer computes, batches writes of newly decoded tokens, and avoids duplicate copies across tiers. It also exposes controls such as lookup, pinning, cleanup, movement and compression, so routers can send a query to the machine that already holds its cache.

With vLLM, it delivered up to 15× throughput on multi-round question answering and document analysis, 3.7–6.8× lower time to first token on real company traces, and much higher loading bandwidth than vLLM's built-in offloading. The paper ends with lessons from production users, including one that matters to anyone designing prompts: truncating long contexts to the most recent tokens cut a real trace's prefix cache hit rate from about 85% to 45%.

Long documents, RAG with repeated sources, coding assistants with growing histories and multi-turn chat all reprocess the same tokens again and again. Each repeat costs prefill compute and time to first token. A KV cache layer turns that repeated work into a lookup.

Reuse is larger than people expect. The authors report that customers did not expect high hit rates, one saw about 50% in production, because reuse comes not only from fixed system prompts but from conversation histories and frequently retrieved documents.

How you handle long context decides whether caching works. Sliding-window truncation, keeping only the most recent tokens, changes the start of the prompt on every turn, so nothing matches the cache. Summarizing old turns into a stable block, or keeping a fixed head and trimming from the middle, preserves the shared prefix.

Loading can beat recomputing, but check the bandwidth. With fast links, loading a stored cache is quicker than recomputing it at every context length the authors tested. With slow links, loading wins only for very long contexts; at 32 Gbps, only beyond about 256K tokens in their setup. Decide per deployment, and per request if possible.

Remote storage has become viable. Object stores that used to deliver about 100 MB/s now approach 1 GB/s, and users loading caches from their own object store saw 22–32% lower time to first token than full prefill.

Routing should know where caches live. With several replicas, sending a request to the replica holding its prefix is the difference between a hit and a recompute.