Reflexion: Language Agents with Verbal Reinforcement Learning

Language agents built on ReAct and similar methods act, observe and act again, but they do not learn from one attempt to the next. If an agent picks up the wrong pan on its first try, it is just as likely to do it on the second. The standard machine-learning answer, reinforcement learning, would update the model's weights from rewards, which takes many samples and a fine-tuning budget that was out of reach for GPT-3.5- and GPT-4-sized models.

Reflexion, from Noah Shinn and colleagues at Northeastern, MIT and Princeton (including ReAct's Shunyu Yao), does the learning in text instead. Three model roles work in a loop. An Actor attempts the task. An Evaluator scores the attempt: exact match against the answer, a hand-written heuristic, an LLM judgment, or unit tests the agent wrote itself. A Self-reflection model reads the failed attempt and its score and writes a short lesson in plain language, such as which action went wrong and what to do instead. The lesson goes into a memory of the last one to three lessons, which is added to the Actor's prompt on the next try. The loop repeats until the evaluator says pass or the trial budget runs out.

The results were strong for 2023. On HumanEval, GPT-4 with Reflexion reached 91% pass@1, against 80.1% for GPT-4 alone. On ALFWorld, text-based household tasks, the agent solved 130 of 134 within 12 trials. On 100 HotPotQA questions, success rose by about 20 points over the trials.

The paper also showed where it fails. On MBPP Python, the agent's self-written tests let wrong code through 16.3% of the time, and Reflexion scored 77.1%, below GPT-4's 80.1% without it.

The pattern in this paper, try, check, write down what went wrong, try again, is now everywhere: coding agents that run tests and fix failures, extraction pipelines that validate output and re-prompt, support agents that retry a failed API call with a corrected argument. Reflexion is the clearest early statement of it, and of its one hard dependency.

The check decides everything. The loop stops when the evaluator says pass. If the evaluator passes wrong answers, the agent ships them confidently; if it fails right answers, the agent may edit working code into broken code. In the paper, the same model and loop gained 11 points on one benchmark and lost 3 on another because of test quality. Before you add a reflection loop, measure how often your check is wrong.

Some reported gains used the answer key. On HotPotQA, the evaluator between trials was exact match against the gold answer, so the agent was told when it was wrong. In ALFWorld, the environment reported success. In production you rarely have either. The programming results are the ones that hold without an answer key, because there the agent used only tests it wrote itself.

It costs attempts. Every reflection is another model call and another attempt. The 2024 paper AI Agents That Matter found that Reflexion cost over 50% more than a simple retry baseline with rising temperature on HumanEval, at similar accuracy. Price the loop before you promise it.

The lessons are readable. Unlike weight updates, a reflection is text a person can audit, which helps debugging and review.