CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion

A RAG prompt stacks several retrieved chunks in front of the question. The same chunks keep coming back across queries, so it is tempting to compute each chunk's KV cache once and reuse it. Two existing ways do that, and both fall short. Prefix caching (vLLM, SGLang) reuses a cache only when it is the prompt's prefix, so with six chunks it saves the first one at best. Full KV reuse (PromptCache) concatenates each chunk's separately computed cache with its positions corrected. It is fast, but each chunk's cache was computed without seeing the chunks before it, so the cross-attention between chunks is missing. The paper's example: one chunk gives Messi's World Cup goals, another Ronaldo's, and full reuse can't compare them.

CacheBlend reuses every chunk's cache and repairs it. On each layer it recomputes the KV of a small share of tokens, the ones whose cached KV deviates most from what a full prefill would produce. Finding them exactly would need the full prefill, so it uses a shortcut: tokens that deviate on one layer tend to deviate on the next. It recomputes all tokens on the first layer, ranks them by deviation, and on later layers recomputes only the top-ranked tokens, narrowing the set slightly each time. About 10% to 15% of tokens carry most of the deviation, and 15% is the default.

The second idea is pipelining. Recomputing a layer's selected tokens can overlap with loading the next layer's cache from CPU memory or SSD; when loading is the slower step, the recompute is free. That makes cheap, large storage (SSD) usable for KV caches.

Across Mistral-7B, Yi-34B and Llama-70B on 2WikiMQA, Musique, SAMSum and MultiNews, time to first token (TTFT) fell 2.2-3.3x against full prefill with quality within about 0.02, and throughput rose 2.8-5x. The code lives in the LMCache repository.

In a customer's RAG deployment, the retrieved context is most of the prompt and most of the prefill time; the question itself is a few dozen tokens. Popular documents, policies and product pages get retrieved again and again, in different combinations. That repeated work is what CacheBlend targets.

Prefix caching does less for RAG than people assume. It helps the shared system prompt and the first chunk. Unless the customer always retrieves the same chunks in the same order, the rest is recomputed every time.

Naive chunk caching is a correctness bug, not just a quality dip. Gluing separately cached chunks together drops the attention between them. Questions that compare or join facts across documents are exactly the ones that break, and they are often the ones the customer cares about.

Where the cache lives is a latency decision. CacheBlend's pipelining lets a slower, cheaper tier serve the cache without adding delay, up to the point where loading becomes the bottleneck. That turns "can we afford to cache this corpus" into arithmetic you can do in an afternoon.