The case for 4-bit precision: k-bit Inference Scaling Laws

Suppose a GPU has room for 30 GB of weights. That holds a 60-billion-parameter model at 4 bits per weight or a 30-billion-parameter model at 8 bits. Which is better? Before this paper, quantization studies asked how much a given model loses when quantized. Dettmers and Zettlemoyer asked the budgeting question instead: for a fixed total number of bits in the model, what precision gives the best accuracy?

They framed it as a scaling law. For five model families (OPT, Pythia/NeoX, GPT-2, BLOOM and BLOOMZ) from 19 million to 176 billion parameters, they quantized weights to every precision from 3 to 8 bits plus 16-bit, measured mean zero-shot accuracy on LAMBADA, PIQA, HellaSwag and WinoGrande, and plotted accuracy against total model bits. The quantization was zero-shot: no calibration data, just a fixed rule applied to the weights, with activations left in 16-bit.

The answer was the same everywhere. Reducing precision from 16 to 4 bits, while spending the saved bits on more parameters, improves accuracy at every budget. At 3 bits the curve turns down; OPT and Pythia become unstable, near chance on their largest models. 4-bit is the best bit-for-bit precision.

Then, across more than 35,000 experiments, they tested what makes low-bit quantization better. Two things helped. Small block size: quantizing groups of 64 weights each with its own scale costs 0.25 extra bits per weight but improved 4-bit about as much as moving to 5 bits. Data type: float and quantile formats beat plain integers. Outlier-aware tricks that rescued 3-bit models did not beat simply using 4-bit with more parameters. The recommendation: use 4 bits with small blocks and a float-like type, and if you need a smaller or larger model, change the parameter count, not the precision.

This paper is the reason "use the biggest model that fits in 4-bit" became standard advice, and it gives you the reasoning to defend that choice.

Spend memory on parameters, not on precision. When a customer's hardware is fixed, compare candidates by total weight bytes. A 4-bit model twice the size of an 8-bit one typically wins in this paper's measurements. The exception the authors give: a GPU that can hold a model at 5 bits but not the next size up at 4 bits; then use the 5 bits.

Block size and data type are cheap settings that matter. A 4-bit format with blocks of 64 and a float or normal-quantile data type spends a quarter-bit per weight on scales and recovers much of the quantization loss. Formats such as NF4 in QLoRA and group-128 settings in GPTQ and AWQ follow this logic.

Going below 4 bits needs more than rounding. Zero-shot 3-bit quantization breaks some model families. Methods that use calibration data, like GPTQ, can do better at 2 to 3 bits, and the paper shows 2-bit GPTQ with small blocks beating its own 3-bit float. Treat sub-4-bit as an experiment, not a default.