Language Modeling Is Compression

Shannon's source coding theorem ties prediction to compression: if a model assigns probability p to the next symbol, an ideal code spends -log2 p bits on it. Arithmetic coding achieves this almost exactly. So a better predictor is a better compressor, and minimizing a language model's log loss is the same as minimizing the length of a compressed file. The reverse also holds: any compressor defines a predictor, because the symbol that compresses best next is the one it considers most likely.

The authors use this view to study large models. They chunk 1 GB each of English Wikipedia (enwik9), grayscale ImageNet patches and LibriSpeech audio into 2,048-byte pieces, compress each piece with arithmetic coding driven by a model, and compare with gzip, LZMA2, PNG and FLAC. Models include small Transformers trained on enwik8, Llama 2 7B and Chinchilla 1B, 7B and 70B.

The large models, trained mostly on text, compressed all three modalities better than the classical compressors on the same chunks, including image patches and audio they were not trained on. But a decompressor needs the model too. Counting each parameter as 2 bytes, a 70-billion-parameter model adds 140 GB to the cost of compressing 1 GB, and its "adjusted" rate becomes 14,008%. Small Transformers trained on enwik8 show the tradeoff directly: each dataset size has an optimal model size, after which bigger models compress worse once their size is counted.

The paper adds three observations. Tokenizers act as a first compression step: larger vocabularies shorten sequences, but small models compressed better with simpler ones. In-context compression is limited by context length. And a compressor such as gzip can generate text by sampling the continuation it compresses best, though the samples are poor. The headline sentence of the abstract cites 43.4% for images and 16.4% for audio, numbers that do not appear in the current version's Table 1.

Log loss is bits. Perplexity and cross-entropy, the numbers on model cards and training curves, are compression rates in disguise. Bits per byte is the most comparable form across tokenizers, which is why it appears in evaluations of models with different vocabularies.

General models are general predictors. A text model compressing image patches better than PNG shows how much generic pattern-matching sits inside large models. It is also a reason to test them on unexpected data types in a customer's domain before ruling them out.

Counting the model changes the answer. The raw rates look magical; the adjusted ones are terrible for 1 GB. The same logic applies to customer decisions: a large model's quality has to be weighed against its serving cost and footprint for the volume at hand, and the best size depends on how much work it will do.

Context length is a hard limit. In-context compression can only use what fits in the window; chunking cost gzip 15 points in our run because each chunk starts with no history. Large documents processed in chunks lose cross-chunk information the same way.

Compression-based checks are useful tools. Compression ratios can flag duplicated or templated data, detect unusual inputs, or give a quick, model-free similarity measure between documents.