AFlow: Automating Agentic Workflow Generation

Most LLM applications that work well are workflows: several model calls in a fixed arrangement, such as generating five answers and voting, or writing a solution and checking it with code. People design these by hand for each task. Jiayi Zhang, Jinyu Xiang and colleagues at DeepWisdom and several universities built AFlow to search for them automatically. The paper appeared at ICLR 2025.

AFlow writes each workflow as a short Python program. Its building blocks are nodes, single LLM calls defined by a model, prompt, temperature and output format, and operators, hand-written combinations of nodes for common patterns: Generate, Format, Review and Revise, Ensemble (a self-consistency vote), Test, Programmer, and a Custom node. To keep the search manageable it fixes the model, temperature and format, and searches over prompts and code.

The search is a variant of Monte Carlo tree search in which every tree node is a complete workflow. Each round it selects a workflow, mixing a uniform choice with a preference for high scores and always allowing a restart from the blank template. An LLM optimizer (Claude 3.5 Sonnet) expands it with one change, using a log of earlier edits to that workflow, whether they helped, and logs of its predictions against the expected outputs. The new workflow is evaluated five times on a validation set, and the result is backpropagated into the experience log. After 20 rounds, the best workflow is tested.

With GPT-4o-mini executing, the found workflows beat every hand-designed baseline on all six benchmarks (HotpotQA, DROP, HumanEval, MBPP, GSM8K and a level-5 MATH subset), averaging 80.3 against 76.0 for the best. Workflows found with one model mostly transferred to others, and cheap models running them matched or beat GPT-4o on HumanEval at a small fraction of its cost.

You will build workflows by hand for customers: retrieval, then draft, then check, then format. AFlow is a credible way to search for a better arrangement once you have a scored dataset, and its results line up with what practitioners find by hand.

The winners are familiar. On GSM8K the search settled on five samples, a vote, and a code check. That is self-consistency plus a programmatic verifier. The operator set already contained those ideas. Expect a search like this to recombine your building blocks well, not to invent new ones.

You need an evaluation function first. AFlow works on tasks with a numeric score: exact answers, unit tests, F1. If the customer's task has no automatic score, the search has nothing to climb.

A small validation set will fool you. Picking the best of twenty candidates on a small set inflates the winner's score. Our simulation below shows how much, and how to spend the evaluation budget instead.

Count the search, not just the result. The paper reports what the found workflows cost to run, not what the search cost. Twenty rounds, each run five times on a validation set, plus an optimizer model, is a real bill.