IN THIS BREAKDOWN · 10 SECTIONS
You can read 17% of this breakdown.
The paper in one minute
By 2021 the largest dense language models had grown from GPT-3's 175 billion parameters to Megatron-Turing NLG's 530 billion, yet nearly all of them were trained on about 300 billion tokens. That followed Kaplan et al. (2020), whose scaling laws said most extra compute should buy a bigger model. DeepMind asked the same question again: for a fixed FLOP budget, how should you trade model size against training tokens?
They trained over 400 Transformers from 70 million to over 16 billion parameters on 5 to 500 billion tokens and estimated the answer three ways. Approach 1 trains each model size with four schedule lengths and takes the lowest loss reached at each FLOP count. Approach 2 fixes nine FLOP budgets, varies model size within each, and fits a parabola to find the best size. Approach 3 fits a single loss function, L(N, D) = E + A/N^α + B/D^β, to all the runs.
The three agree closely. The optimal model size grows with compute as C^0.46 to C^0.50, and tokens as C^0.50 to C^0.54. Model size and data should scale together. On the frontier, a compute-optimal model sees about 20 tokens per parameter. By that standard, Gopher (280B on 300B tokens) was four times too big.
To test it, DeepMind trained Chinchilla: 70 billion parameters on 1.4 trillion tokens, the same compute as Gopher. It beat Gopher, GPT-3, Jurassic-1 and MT-NLG 530B on almost every evaluation, including 67.6% on MMLU against Gopher's 60.0%, and it costs a quarter as much to run.
Research notes: the problem, approach and results
exponent a in N_opt ∝ C^a across the three approaches, against Kaplan's 0.73 (Table 2)
training tokens per parameter on the compute-optimal frontier in Approach 1 (Table 3: 1B on 20.2B tokens)
5-shot MMLU, Chinchilla 70B against Gopher 280B at the same training compute (Table 6)
Following Kaplan et al. (2020), the largest language models of 2021 grew to 175-530 billion parameters while staying at roughly 300 billion training tokens. Nobody had checked whether that split of the compute budget was right with schedules tuned to each run's length.
Train over 400 models from 70 million to over 16 billion parameters on 5 to 500 billion tokens, with the learning-rate schedule matched to each run's length. Estimate the compute-optimal model size three ways: the envelope of training curves, the minimum of fixed-FLOP profiles, and a fitted loss function L = E + A/N^α + B/D^β.
All three approaches put the exponents near 0.5: double the model, double the tokens. At Gopher's 5.76 × 10^23 FLOPs the optimum was 40-70B parameters. Chinchilla, 70B on 1.4T tokens, reached 67.6% on MMLU against Gopher's 60.0%, and is four times cheaper to serve.
Why it matters to an FDE
Chinchilla is the reason a 7-billion-parameter model trained on trillions of tokens can beat a much larger model from the year before, and the reason parameter count is a poor proxy for quality when a customer compares vendors.
Training tokens matter as much as parameters. When you evaluate open-weight models, ask how many tokens each saw. A model far below about 20 tokens per parameter left quality on the table for its compute; many recent small models are trained far beyond it on purpose.
Compute-optimal is not deployment-optimal. The paper optimizes training FLOPs only. You pay for inference on every request, and a smaller model serves more cheaply. The authors point this out themselves: Chinchilla matches or beats Gopher while needing a quarter of the memory and compute per token. If a model will serve billions of tokens, it can be worth training a smaller one far past the Chinchilla point.
Data becomes the constraint. Equal scaling means a 10x bigger budget needs about 3x more tokens. The paper flags dataset collection and quality, and train/test overlap, as the problems that follow.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The equal-scaling rule, the fitted loss, what changed from Kaplan, and two misreadings
Free account - 04How it works
The three estimation approaches, the parametric fit, the efficient frontier and the Chinchilla run
Premium - 05The numbers that matter
Table 2's exponents, Table 3's token budgets, and Chinchilla against Gopher
Premium - 06Try it yourself
Recompute the frontier from the fitted loss, audit the FLOP tables, and add serving cost
Premium - 07Where it breaks
Two large runs, curvature, one epoch, training-only cost and the slips
Premium - 08Using it on a deployment
Reading model cards, choosing between sizes, and when to train past the Chinchilla point
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium