Scaling Laws for Neural Language Models

Before this paper, deciding how big a language model to train was mostly judgment. Kaplan and colleagues at OpenAI trained a large grid of decoder-only Transformers on WebText2, a 22-billion-token web scrape, and measured test loss as they varied the number of non-embedding parameters N (768 to 1.5 billion), the dataset size D (22 million to 23 billion tokens), the compute C, the batch size, and the model's shape.

Three results came out. First, loss is a power law in each of N, D and C whenever the other two are not the bottleneck: L(N) = (8.8 × 10^13 / N)^0.076, L(D) = (5.4 × 10^13 / D)^0.095, and against optimally allocated compute, L ∝ C^−0.050. The trends held over six to eight orders of magnitude with no sign of bending. Second, shape barely matters: at a fixed parameter count, a model 6 layers deep and 4,288 wide came within 3% of the loss of a 48-layer, 1,600-wide one. Counting embedding parameters muddied the trends; excluding them gave clean lines. Third, larger models are more sample-efficient: they reach a given loss in fewer steps and fewer tokens.

From those fits the authors solved for the best use of a fixed compute budget. Their answer was to grow the model fastest, as C^0.73, and the data slowly, as C^0.27, stopping well before convergence. Many large models that followed, GPT-3 among them, were trained on about 300 billion tokens in line with that rule. Two years later Chinchilla showed the data side had been badly underweighted. The power-law method survived; the budget rule did not.

You will rarely pretrain a model, but this paper hands you the arithmetic that sits under every model-sizing conversation.

The 6N rule prices training. Training costs about 6 floating-point operations per non-embedding parameter per token (2 for the forward pass, 4 for the backward). A 7-billion-parameter model on a trillion tokens is about 4.2 × 10^22 FLOPs. Convert that to GPU-days with a utilization figure you have measured and you have a defensible estimate for a customer's continued-pretraining plan before anyone books hardware. Forward-only inference is the 2N part: about 2 FLOPs per parameter per generated token.

Smooth curves let small runs predict large ones. The paper's lasting method is to train a ladder of small models, fit the curve, and extrapolate. Later scaling work, Chinchilla included, kept that method while changing the conclusions, and the same logic justifies running a fine-tuning or data-mix experiment at small scale before spending on the large one.

Extrapolation is only as good as the bottleneck you modeled. The paper's own later sections show the fits fail when one factor binds, and its budget advice reflected a training setup, not a law of nature. When someone shows you a straight line on log-log axes, ask what was held fixed.