CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving

Caching a long context's KV cache saves prefill the next time the context is used: a chat history, a contract, a codebase file. The catch the paper points at is that the cache rarely stays in the same GPU's memory. It gets offloaded, stored on another server, or needed by a different GPU, and it is big. The paper's example: an 80,000-token annual report on a 34B model produces 19 GB of KV, about the size of the model. Between regular cloud servers, at single-digit Gbps, fetching that can take longer than computing it from the text.

CacheGen shrinks the cache for transmission rather than for GPU memory. Since the cache only has to be rebuilt into tensors on arrival, it can be encoded as a bitstream, much like video. Three observations drive the encoder. Within a channel, nearby tokens have similar values, so it stores one anchor token per group of 10 at 8 bits and the rest as differences. Errors in early layers hurt more than errors in deep layers, so deeper layers get coarser quantization. Values grouped by channel and layer have lower entropy than values grouped by token, so arithmetic coding uses a probability table per channel and layer, profiled once per model.

The second half is streaming. Each context is split into 1,500-token chunks, each encoded offline at several levels. When a request arrives, chunks are sent one at a time; for each, CacheGen measures the throughput of the previous chunk and picks the least lossy option that still meets the time-to-first-token (TTFT) target, including sending the raw text and recomputing that chunk. Decoding runs on the GPU and overlaps the next chunk's transfer.

On Mistral-7B, Llama-34B and Llama-70B over 662 long contexts, CacheGen's KV was 3.5-4.3x smaller than uniformly quantized KV at similar quality. At 3 Gbps, TTFT was 3.2-3.7x lower than the quantization baseline and 3.1-4.7x lower than sending text. Under fluctuating bandwidth with a 1 s target, misses dropped from 81% to 8%.

Once a customer's contexts outgrow GPU memory (long chat histories, document assistants, many tenants), cached KV lives in host memory, on SSD or on a storage service. At that point network bandwidth, not GPU compute, decides whether the cache helps.

Do the transfer arithmetic before promising a cache win. KV size per token is fixed by the model's shape; divide by the link speed and compare with prefill time. On a 10 Gbps link a multi-gigabyte cache may be slower than prefill.

Transmission-time compression is a different lever from GPU-memory compression. KV quantization for memory must keep tensors usable by the attention kernel; a codec for the wire can use entropy coding and deltas and decode on arrival. The two stack.

Treat cache fetch like a video stream with a deadline. Bandwidth inside a cloud region varies. Choosing quality per chunk against a TTFT target, with text as a fallback, keeps latency predictable instead of hoping the link holds.