GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

The simplest way to compress a model's weights is to round each one to the nearest value on a 4- or 3-bit grid. For 8 bits, that works. For 4 and 3 bits on very large models, it does not: rounding OPT-175B to 4 bits made its perplexity worse than a full-precision model ten times smaller, and at 3 bits the model produced nonsense. Methods that rounded more carefully existed, but they were built for vision models of around 100 million parameters and would have taken far too long at 175 billion.

GPTQ makes careful rounding fast enough. It works one linear layer at a time, with a goal: after quantizing the layer's weights W, the layer's outputs on a small calibration set X should change as little as possible. That is a least-squares problem whose curvature is the matrix H = 2XXᵀ, computed from the calibration inputs. GPTQ rounds the weights one column at a time, in the same order for every row, and after each column it nudges all the not-yet-rounded weights to cancel the error that column introduced. Correlated inputs mean another weight can often absorb most of the damage.

Three engineering changes make it scale. A fixed column order lets every row share one inverse-Hessian computation. Updates are batched in blocks of 128 columns so the GPU is busy rather than waiting on memory. And the needed rows of the inverse Hessian are taken from a Cholesky decomposition, which stays numerically stable where repeated matrix updates did not.

The results: OPT-175B and BLOOM-176B quantized in about four GPU-hours with 128 sequences of generic web text. At 4 bits, perplexity barely moved; at 3 bits it rose a fraction of a point. The 3-bit OPT-175B fit in a single 80 GB A100, and custom kernels that read 3-bit weights and dequantize them on the fly made generation 3.24 times faster than fp16 spread over five GPUs.

GPTQ is one of the formats you will meet most when someone downloads a "4-bit" open model, and its lessons apply to any weight-only quantization.

Fewer GPUs, faster tokens at batch 1. Generation at small batch sizes is limited by how fast weights stream from memory. A 3- or 4-bit model reads a quarter or less of the bytes per token, so it can be both cheaper to host and faster per user, unlike int8 with outlier handling, which mainly saves memory.

Calibration data is small but not free. GPTQ needs a few hundred calibration sequences and an hour or a few on one GPU per model. The paper uses generic web text and reports results on standard benchmarks. If a customer's domain is unusual, quantize with representative domain text and check.

4 bits is the safe default; 3 bits needs testing. At 4 bits the paper's largest models lost under 0.25 perplexity points; at 3 bits up to about one point on some tests. Smaller models lose more. Test on the customer's task before going below 4 bits.