Large Language Models Cannot Self-Correct Reasoning Yet

By 2023, papers such as Reflexion and RCI reported that language models could improve their reasoning by reviewing their own answers and trying again. Jie Huang, Xinyun Chen, Denny Zhou and colleagues at Google DeepMind and UIUC asked a pointed question: if a model can recognize that its answer is wrong, why does it give the wrong answer in the first place? Their paper, presented at ICLR 2024, retests the claim under the conditions a deployment actually has.

They define intrinsic self-correction: the model generates an answer, is prompted to review it and report problems, and then answers again, up to two rounds, with no external feedback. Nothing tells it whether the answer was right. This differs from the earlier setups, which used the oracle label, the known correct answer, to stop the loop as soon as the answer was right.

With oracle labels, self-correction looked strong: GPT-3.5 rose from 75.9% to 84.3% on GSM8K and from 75.8% to 89.7% on CommonSenseQA. Without them, every model they tested got worse. GPT-3.5 on GSM8K fell to 74.7%, and on CommonSenseQA collapsed to 41.8%. GPT-4 fell from 95.5% to 89.0% on GSM8K. Llama-2 70B dropped from 62.0% to 36.5%. Trying three different feedback prompts did not change the picture.

The paper then checks two other common comparisons. Multi-agent debate (several copies of the model critiquing each other) was no better than simple majority voting over the same number of answers, and worse at six and nine responses. And on Self-Refine's constrained generation task, a clearer initial prompt alone scored 81.8%, above the 67.0% the original paper reported after self-correction; adding self-correction on top lowered it to 75.1%.

"Have the model check its own work" is one of the most common suggestions in customer design sessions. This paper is the evidence to bring to that conversation.

Without an outside signal, a review step can make reasoning worse. On math and multiple-choice questions, models changed right answers to wrong ones at least as often as the reverse. Adding a self-check is not free insurance.

An oracle in the loop is not a deployable method. If an evaluation stops revising when the answer is correct, it is using the answer key. Ask how every self-correction result decided when to stop.

Bring a real verifier. Unit tests, a calculator, a schema validator, a retrieval check, a human. The authors point to code with tests as the case where correction works, because execution tells the model what is wrong.

Compare at equal cost. A two-round self-correction uses five calls per question. Five independent answers and a vote is the fair baseline, and in this paper voting won.

Put the requirements in the first prompt. If the critique prompt contains instructions the first prompt lacked, the "improvement" comes from the instructions. Write the best first prompt you can before judging a revision step.