Self-Refine: Iterative Refinement with Self-Feedback

People rarely get a piece of writing or code right on the first try; they reread it, notice problems, and revise. Aman Madaan, Niket Tandon, Peter Clark and colleagues at CMU, the Allen Institute for AI and elsewhere asked whether a language model can do the same with no training, using only prompts. Their method, Self-Refine, presented at NeurIPS 2023, uses one model in three roles.

First it generates an output from a few-shot prompt. Then it writes feedback on that output, using a prompt whose examples show feedback that is actionable (it proposes a concrete change) and specific (it names the part to change): "This code is slow as it uses a for loop which is brute force. A better approach is to use the formula n(n+1)/2." Then it refines: rewrites the output given the feedback and the full history of earlier outputs and feedback. The loop repeats up to four times, or stops when the feedback includes a stop signal, such as a quality score above a threshold.

They tested seven tasks: making dialogue responses more engaging, speeding up programs, improving variable names, math word problems (GSM8K), reversing the sentiment of reviews, generating acronyms, and writing a sentence that uses 20 to 30 given keywords. With GPT-3.5, ChatGPT and GPT-4, Self-Refine improved six of the seven, often by a lot. GPT-4's keyword coverage tripled from 15% to 45%. Math did not move: the model's feedback almost never noticed its own errors.

The analysis is as useful as the results. Specific feedback beat generic feedback, which beat none. Most of the gain came in the first round or two. And when refinement failed, the feedback was almost always to blame.

A draft, critique, rewrite loop is easy to add to almost any generation step, and customers often suggest it. This paper tells you where it pays.

It works when the model can see the problem. Missing keywords, a cold tone, a nested loop, single-letter variable names: things a reader spots and fixes. It does not work when spotting the problem is the hard part, as with a subtle arithmetic slip.

The feedback step is the bottleneck. In the authors' review of failures, 94% came from feedback that pointed at the wrong place or suggested the wrong fix. Spend effort on the critique prompt, and give it external signals (test results, a validator, a checklist) whenever you can.

An outside signal changes everything. On math, telling the model only which answers were wrong raised GPT-3.5 by 4.8 points where its own feedback had added nothing. If you have a check in code, use it to decide when to refine.

Read preference numbers carefully. Half the tasks were scored by asking GPT-4 which of two outputs is better. A large win-rate gap can come from small but consistent improvements, so do not read it as a quality percentage.