Language Models are Few-Shot Learners

In 2020 the standard recipe for a language task was to take a pre-trained model and fine-tune it on thousands of labeled examples of that one task. A new task meant a new dataset and a new model. People do not work that way. Show a person one or two examples and a sentence of instructions and they usually get it.

OpenAI asked whether a big enough language model could do the same. They trained GPT-3, a decoder-only Transformer with 175 billion parameters, on 300 billion tokens of web text, books and Wikipedia, and then never fine-tuned it. Every task was given to the model as text: an instruction, sometimes a handful of solved examples, then the case to solve. They called this in-context learning, and named three settings by how many examples the prompt contains: zero-shot, one-shot and few-shot.

The results moved the field. On closed-book TriviaQA, GPT-3 went from 64.3% zero-shot to 71.2% few-shot, the best published result for that setting in 2020, ahead of fine-tuned systems. On SuperGLUE, 32 examples per task in the prompt beat a BERT-Large model fine-tuned on 125,000. Small models could barely do any of this; the ability grew with size.

The paper is also unusually frank about where it failed, and those failures are as useful as the wins. Almost every LLM product since starts from its premise: describe the task in the prompt.

This paper defines the first decision on almost every engagement: do we prompt, or do we train? It names the spectrum. Fine-tuning changes weights with thousands of examples. Few-shot puts 10 to 100 examples in the prompt. Zero-shot gives only an instruction. It also states the trade-off plainly: few-shot needs far less data, but it was then "much worse than state-of-the-art fine-tuned models" on many tasks.

Two findings matter more on a deployment than the headline scores.

Comparing two texts was a blind spot. GPT-3 scored 49.4% on WiC, a two-way choice where chance is 50%, and was near chance on ANLI. Both ask whether two pieces of text mean the same thing or whether one implies the other. That is exactly the shape of many enterprise asks: does this clause match our policy, does this invoice match the purchase order. Modern models are far stronger, but this is still the task type to test hardest before promising anything.

More examples is not always better. On the LAMBADA benchmark, one-shot accuracy (72.5%) came in below zero-shot (76.2%), though the paper notes zero-shot used a different prompt format. Prompt design is an experiment, not a formula, and each example you add costs tokens on every request.