Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

GPT-3 showed that a few examples in a prompt could teach a model a task. But on problems that need several steps, such as a math word problem, few-shot prompting did poorly, and making the model bigger did not help much. The examples showed the question and the answer, and the model was expected to jump straight from one to the other.

The Google team changed one thing: they rewrote the examples to include the working. Instead of "A: The answer is 11", the example says "Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11." The model, seeing eight examples like that, writes out its own steps before answering. They called this a chain of thought.

On GSM8K, a benchmark of grade-school math word problems, PaLM 540B went from 17.9% correct to 56.9%, beating the previous best, a GPT-3 model fine-tuned on the training set with a separate verifier. Similar gains showed up on commonsense and symbolic tasks.

Two findings travel further than the headline. The trick only works on large models: below roughly 100 billion parameters, models wrote fluent nonsense and did worse. And the paper's ablations show that what helps is the reasoning in language, not just extra tokens or a written equation. This is the idea under every "think step by step" prompt and every reasoning model since.

Chain of thought is now built into the models: reasoning models do it without being asked, and many APIs expose a setting for how much they "think". The paper is still the best guide to when that thinking is worth paying for.

It helps on multi-step problems and not much else. The paper's own breakdown shows gains were largest on the hardest problems and "either negative or very small" on single-step ones. A customer's pipeline that routes, extracts or classifies may get slower and more expensive with reasoning turned on and no more accurate. Measure it on their task before switching it on everywhere.

Reasoning costs tokens. Each chain adds output tokens, which are slower and usually pricier than input tokens. A worked answer that is ten times longer than a bare answer is roughly ten times the generation time for that request.

A plausible chain is not a correct one. The authors say plainly that there is "no guarantee of correct reasoning paths". In their own error review, a wrong answer often came with a chain that was nearly right. For any workflow where the reasoning is shown to a user or an auditor, the steps need checking, not just the final answer.