LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Quantizing a model's weights from 16-bit to 8-bit integers halves the memory they take. For small transformers this worked; for multi-billion-parameter models, every 8-bit method tried before this paper degraded quality, and the larger the model, the worse it got. Dettmers and colleagues found out why and fixed it.

The cause is a property of large transformers they call emergent outlier features. In a model's hidden states, a few specific feature dimensions carry values up to 20 times larger than the others. They appear in some layers in smaller models and, at around 6.7 billion parameters, show up in the same few dimensions in every layer and for most tokens. In a 6.7B model with 2,048-token sequences there are about 150,000 such outliers per sequence, but they sit in just 6 dimensions. They matter: zeroing them halves the model's top-1 attention probability and makes perplexity 6 to 10 times worse. Zeroing 6 random dimensions barely changes anything.

Ordinary int8 quantization scales a whole tensor (or row) by its largest value. One outlier of 60 forces every ordinary value, which sits within about ±3.5, into a handful of the 255 integer levels, and precision collapses.

LLM.int8() has two parts. Vector-wise quantization gives every row of the input and every column of the weights its own scaling constant. Mixed-precision decomposition pulls out any feature column containing a value of magnitude 6 or more and multiplies those few columns in 16-bit, while more than 99.9% of the values go through int8. The method needs no retraining or calibration data: load a 16-bit checkpoint, convert, and run. It held 16-bit accuracy up to 175B parameters and was released as the bitsandbytes library with a Hugging Face integration.

This is the method behind load_in_8bit=True in Hugging Face Transformers, and its findings explain why quantizing large models takes more than rounding.

Memory is the usual blocker, and this halves it. A model that does not fit on a customer's GPUs in 16-bit often fits in int8: the paper's Table 2 runs OPT-175B on eight 24 GB RTX 3090s, which could only hold OPT-66B in 16-bit. When the alternative is buying hardware or using a smaller model, int8 is a quick first try.

It is not a speedup. The paper's own benchmarks show the decomposition adds overhead. For models of 6.7B and smaller, LLM.int8() matrix multiplication is slower than 16-bit; on BLOOM-176B, per-token latency rose about 5% on the same GPUs. Use it to fit a model, not to make it faster.

Outliers are why quantization needs care. The same few large features cause trouble for 4-bit and activation quantization methods that came after. When a quantized model loses accuracy, a handful of extreme dimensions is the first suspect.