IN THIS BREAKDOWN · 10 SECTIONS
You can read 20% of this breakdown.
The paper in one minute
Scaling laws say that a language model's pretraining loss falls predictably as compute grows. This paper collects cases where task performance does not: a model family scores at chance for several orders of magnitude of compute, then rises sharply.
The definition is simple. An ability is emergent if it is not present in smaller models but is present in larger ones, which means it cannot be predicted by extrapolating smaller models. Scale is measured mainly in training FLOPs, with parameter counts in the appendix.
The first group of examples is few-shot prompting. On BIG-Bench arithmetic, GPT-3 stays near zero until 2 x 10^22 FLOPs (13B parameters) and LaMDA until 10^23 (68B). MMLU stays at chance for models below about 10^22 FLOPs and rises past random at 3 to 5 x 10^23. Word in Context stays at chance through GPT-3 175B and Chinchilla and rises only with PaLM 540B. The second group is methods: chain-of-thought prompting, instruction tuning, scratchpads and a calibration technique each hurt or do nothing for small models and help only past a scale.
The authors offer partial explanations and say the full one is unknown. Exact-match metrics hide gradual progress, but classification tasks also emerge. Cross-entropy on the targets improves for small models while accuracy stays flat. Better data or objectives can bring an ability to smaller models: PaLM 62B was above random on 14 BIG-Bench tasks where GPT-3 175B and LaMDA 137B were not.
Research notes: the problem, approach and results
training FLOPs (about 100B parameters) where chain-of-thought prompting first beat standard prompting on math word problems (Figure 3A)
BIG-Bench tasks where PaLM 62B was above random while LaMDA 137B and GPT-3 175B were near random (Section 5.2)
ratio of Table 1's Chinchilla FLOPs to 6 x parameters x tokens; the other rows match within 2% (our recomputation)
Scaling laws predict that pretraining loss falls smoothly with compute, but performance on particular tasks does not always follow. Some tasks stay at chance across several model sizes and then succeed, which matters for anyone planning what a bigger model will do.
Define an emergent ability as one absent in smaller models and present in larger ones, so it cannot be extrapolated, and survey examples across model families: few-shot tasks from BIG-Bench, MMLU, TruthfulQA and others, plus prompting and fine-tuning methods that only help large models.
The paper lists 22 entries plus many BIG-Bench tasks, with thresholds from about 10^20 to 10^25 training FLOPs, and discusses explanations: metrics without partial credit, multi-step tasks, memorization. It notes that cross-entropy improves smoothly where accuracy does not, and that smaller models trained differently can reach some abilities earlier.
Why it matters to an FDE
A pilot on a small model can understate a large one, and the reverse. If a capability is emergent, testing a cheap model tells you little about whether the expensive one will work. Test the model size you plan to deploy, on the customer's task.
Prompting methods depend on scale. Chain-of-thought hurt or did nothing below about 10^23 FLOPs in the paper's data, and instruction tuning hurt models of 8B parameters and below in one study. When a customer moves to a small or distilled model to cut cost, re-test the prompting strategy rather than carrying it over.
Scale is one lever among several. The paper's own examples show data quality and training objective moving thresholds. When choosing between a bigger model and a better-trained smaller one, measure both.
Read "emergent" claims carefully. The companion paper on this site argues that many of these jumps come from metrics and sparse sampling of model sizes. Both papers agree that per-token loss improves smoothly.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The definition, the examples, scale measures, explanations, beyond-scaling cases and two misreadings
Free account - 04How it works
The survey method, scale axes, sources of each example, and the cross-entropy analysis
Premium - 05The numbers that matter
Table 1's thresholds, the method thresholds, MMLU and WiC, and the beyond-scaling cases
Premium - 06Try it yourself
Recompute Table 1's FLOPs, sample a smooth curve at few sizes, and test "above random"
Premium - 07Where it breaks
A survey of other people's curves, few model sizes, metric choice, and the open explanation
Premium - 08Using it on a deployment
Pilot on the target size, re-test prompting when downsizing, and read scaling claims with care
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium