KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

When an LLM serves many requests at once, the KV cache grows with batch size times context length and can outweigh the model itself. It also has to be read from GPU memory for every generated token, so it limits speed as well as capacity. Storing it in fewer bits helps both. The problem is accuracy: with plain round-to-nearest quantization applied per token (the usual way, since tokens arrive one at a time), 4 bits works and 2 bits breaks. On Llama-2-13B, GSM8K falls from 22.67 to 4.55.

The authors looked at what the cache contains. Keys have a few fixed channels with very large values, consistent with earlier findings about activation outliers. Quantizing per token puts a huge channel in every group, so the scale is set by the outlier and the normal channels get crushed. Quantizing per channel (each group is 32 tokens of one channel) keeps each outlier's error in its own channel. Values have no such channels. They are used as a weighted sum over tokens with sparse attention weights, so what matters is that the few attended tokens are accurate; per token grouping keeps each token's error to itself.

Per-channel grouping needs 32 tokens before a group can be formed, and tokens arrive one at a time. KIVI keeps the newest tokens (up to R = 128) in full precision and quantizes them in blocks when the window fills. That window turns out to matter: without it, 2-bit keys per channel and values per token score 12.21 on GSM8K; with it (KIVI-2), 20.77.

KIVI needs no calibration data or fine-tuning. Across Llama-2, Falcon and Mistral, 2-bit KIVI stayed close to 16-bit on standard and long-context tasks (Falcon-7B, which already shares one KV head, needed 4 bits). On Llama-2-7B, peak memory fell 2.6x, batches grew up to 4x and throughput rose 2.35-3.47x on one A100.

For long contexts or high concurrency, the KV cache is what runs out first. How many users fit on one GPU, and therefore cost per request, often comes down to bytes per KV value.

KV quantization is a capacity lever you can pull without retraining. KIVI is plug-in: no calibration set and no fine-tuning, so it can be tried against a customer's own eval in a day.

Ask how the cache is grouped, not just how many bits. Two 2-bit schemes differ by 16 GSM8K points on the same model. When evaluating a serving stack's "2-bit KV" option, the grouping and the recent-token window decide whether it works.

Averages hide the tasks that break. LongBench averages moved about a point, while passage retrieval on Mistral-7B-Instruct-v0.2 dropped 8.5 points and code completion on Llama-3-8B-Instruct dropped 6. If the customer's product is retrieval over long context or code, test that task directly, and consider 4 bits.