Large Language Models are Zero-Shot Reasoners

Chain-of-thought prompting, published a few months earlier, showed that large models solve multi-step problems far better when the prompt contains worked examples whose answers spell out the intermediate steps. That needs someone to write those examples for each task. Kojima and colleagues asked whether the examples were doing the work, or whether the model only needed to be told to reason before answering.

Their method, Zero-shot-CoT, makes two calls to the same model. The first formats the question as "Q: [question] A: Let's think step by step." and generates text greedily, which produces a written reasoning path. The second call feeds back the question, that reasoning and a short answer trigger chosen for the answer type, for example "Therefore, the answer (arabic numerals) is" for arithmetic or "Therefore, among A through E, the answer is" for multiple choice. A simple parser then takes the first number, or the first capital letter, in the reply.

On text-davinci-002, MultiArith accuracy rose from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%. Symbolic tasks moved even more: last-letter concatenation went from 0.2% to 57.6% and coin flip from 12.8% to 91.4%. PaLM 540B showed similar jumps. Easy one-step arithmetic stayed about the same, and two commonsense benchmarks did not improve. The gains appeared only in large models; small models stayed flat with or without the trigger.

Almost every reasoning prompt in production descends from this paper, and three of its findings still shape how you build one.

A reasoning instruction is a cheap first baseline. Before writing few-shot examples for a customer's task, try a plain instruction to reason first. On MultiArith the single sentence recovered about 81% of what eight hand-written chain-of-thought examples gained, and on GSM8K about 79%. Examples still won, so treat the trigger as the baseline to beat, not the final prompt.

Separate the reasoning from the answer. The method works because the second call asks for the answer in a fixed format after the model has finished thinking. Asking for free reasoning and a parseable answer in one breath is where many production prompts fail. The same split can be done today with a structured output field for the final answer.

The parser is part of the score. In Table 1, plain zero-shot scored 12.7% on StrategyQA and 12.8% on coin flip, both yes-or-no tasks where guessing gives about 50%. With a generic "The answer is" trigger instead, the same model scored 54.3% and 53.8%. Before you conclude that a model cannot do a task, check how many answers your parser failed to find.