SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Pruning sets some of a model's weights to zero so they need not be stored or multiplied. For vision models and small language models, it works well, but only with retraining: prune, then train again so the survivors compensate. Retraining a 175-billion-parameter model is not something most teams can do. The only pruning method cheap enough to apply directly at that scale, dropping the weights with the smallest magnitude, kept OPT-175B intact only up to about 10% sparsity and destroyed it beyond.

SparseGPT, from the authors of GPTQ, does accurate pruning in one shot. Like GPTQ, it works one linear layer at a time with a small calibration set (128 sequences of 2,048 tokens of web text) and asks: which weights can be removed, and how should the remaining ones change, so that the layer's output on those inputs barely moves? That question is a sparse regression problem; solving it exactly for billions of weights is hopeless, so SparseGPT approximates it with the same trick GPTQ uses for rounding. It processes columns left to right; when a weight is removed, its contribution is redistributed onto the columns not yet processed, using a row of the inverse Hessian. Which weights to remove is decided in small blocks of columns, by the classic Optimal Brain Surgeon score w²/[H⁻¹]_jj, so the mask adapts as the weights change.

On OPT-175B and BLOOM-176B, SparseGPT removed half the weights, about 100 billion, with perplexity unchanged or slightly better, and reached 60% with small losses. It also handles 2:4 and 4:8 semi-structured sparsity, the patterns NVIDIA GPUs from Ampere onward can accelerate, and can prune and quantize to 4 bits in the same pass. A 175B model takes about four hours on one A100.

Pruning is the other half of model compression alongside quantization, and this paper is the reason it is practical for large language models without retraining.

Sparsity saves nothing until something uses it. A matrix with half its entries zero stored as an ordinary dense matrix takes the same memory and the same time. Savings need a sparse storage format or kernel: 2:4 sparse tensor cores on NVIDIA GPUs, or a sparse inference engine on CPUs. The paper's own measurements show 1.54x to 1.79x faster layers with 2:4 on GPUs and 1.57x to 2.16x end to end on a CPU engine.

Quantization is usually the first move. For memory, 4-bit quantization saves four times; 50% sparsity with fp16 values saves less once you count the index or mask. SparseGPT's own comparison pits 50% sparsity plus 4-bit against plain 3-bit at the same size. Sparsity is worth it where kernels turn it into compute savings.

Large models tolerate pruning better. The paper finds the bigger the model, the less pruning hurts: at 50%, OPT-2.7B lost about a point of perplexity while OPT-175B gained a little. That points to pruning as a tool for the largest models in a fleet.