AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration

When a model generates text one token at a time, each step reads every weight from memory and does very little arithmetic with it. Generation is memory-bound, so storing weights in 4 bits instead of 16 cuts memory traffic about 4× and, with the right kernels, speeds generation up. The problem is accuracy: simple rounding to 3 or 4 bits hurts, and the best method at the time, GPTQ, reconstructs weights against a calibration set and can overfit to it.

AWQ, from Song Han's lab at MIT with collaborators at SJTU, NVIDIA and Tsinghua, starts from an observation. Not all weights matter equally: keeping just 0.1% to 1% of weight channels in 16-bit while rounding the rest recovers most of the lost accuracy, but only if you pick the right ones. Picking channels with the largest weights does about as well as picking at random. Picking the channels that receive the largest activations works, because those channels process the most important features.

Mixed precision is awkward for hardware, so AWQ protects those channels a different way. Multiply a weight channel by a factor s greater than 1 and divide its input by s: the layer's output is unchanged, but the channel's rounding error becomes smaller relative to its size. AWQ sets s from each channel's average activation magnitude raised to a power α, and finds α with a quick grid search on a small calibration set, with no backpropagation or reconstruction. Every weight stays in low bits.

The method beat plain rounding and GPTQ on LLaMA and Llama-2 models at 3 and 4 bits, held up when the calibration data came from a different domain, and worked for instruction-tuned and multimodal models. The authors also built TinyChat, an inference engine with fused 4-bit kernels, which ran more than 3× faster than the Hugging Face FP16 implementation and put Llama-2-70B on a 64 GB Jetson Orin.

AWQ is one of the standard ways to ship 4-bit open models; many published checkpoints are "AWQ" versions, and serving stacks such as vLLM, TensorRT-LLM and Hugging Face support them. For self-hosted deployments it changes what hardware a customer needs.

Quantization buys memory and speed for generation. A 4-bit model needs about a quarter of the memory for weights, which can mean one GPU instead of two, or a laptop or edge device instead of a server. Because decoding is memory-bound, fewer bytes per weight also means faster tokens at small batch sizes.

Calibration data can matter less than you think, which is good. AWQ only collects activation statistics and does not fit weights to the calibration set, so it was much less sensitive to calibration domain than GPTQ. For customer models in specialized domains, that reduces the risk of a quantized model doing well on generic text and worse on the customer's data.

Still evaluate on the customer's tasks. Perplexity on WikiText is the paper's main measure. Run the customer's evaluation on the quantized model before switching, especially for math, code and long reasoning, where small errors compound.

Weight-only quantization helps decoding most. Under large batches, serving becomes more compute-bound and the speed gain shrinks; memory savings remain. Measure throughput at the customer's real batch sizes.

3-bit is possible; 4-bit is the safe default. At 4 bits AWQ was close to FP16; at 3 bits the gap was noticeable even with AWQ.