IN THIS BREAKDOWN · 10 SECTIONS
You can read 20% of this breakdown.
The paper in one minute
Self-consistency samples several reasoning paths and votes. Multi-agent debate has several model instances argue and then vote. Both combine many outputs, and both papers report that more outputs help. This paper isolates that ingredient.
The method, called Agent Forest after random forests, has two phases. Sampling: send the query N times to one model, or to a method built on it (chain of thought, debate, self-reflection). Voting: score each answer by its total similarity to the others and return the highest. For math and multiple choice, similarity is exact agreement, so this is majority voting. For code, it is BLEU between programs. An "agent" here is one sampled call; the agents do not talk to each other.
Across GSM8K, MATH, MMLU, a chess state-tracking task and HumanEval, with Llama2-13B, Llama2-70B and GPT-3.5-Turbo, accuracy rose as N grew to 40. Llama2-13B with 40 samples beat a single Llama2-70B call on GSM8K (0.59 against 0.54). Voting on top of chain of thought, zero-shot chain of thought, Solo Performance Prompting, debate or reflection improved each of them, and voting alone had the best average rank. The authors then build a synthetic task to show when voting pays: gains grow with the number of reasoning steps, rise then fall with difficulty, and shrink when the correct answer is one of many.
Research notes: the problem, approach and results
Llama2-13B on GSM8K, one call against 40-sample voting; one Llama2-70B call scores 0.54 (Table 2)
GPT-3.5-Turbo with 40 samples against one GPT-4 call on GSM8K (Table 2)
extra GPT-3.5 tokens per additional correct GSM8K answer at 40 samples, from Table 5's per-call count (our calculation)
Multi-agent frameworks, debate and self-consistency all combine several model outputs, and each paper credits its own structure. Nobody had measured how much accuracy comes simply from asking more times and voting, across tasks and models.
Sample N answers from the same model, alone or wrapped in another method such as chain of thought or debate, and pick the answer with the highest total similarity to the others. Scale N to 40 on five benchmarks and three models, then study which task properties make voting pay.
Accuracy rose with N on every task and model, by 12 to 24 points on GSM8K and 6 to 10 on MATH. Voting added gains on top of every method tested and alone ranked best on average. Gains were larger for weaker models and harder tasks, until tasks became too hard.
Why it matters to an FDE
It is the baseline any multi-agent design must beat. Before a customer builds a debate, critique or role-play framework, run the same model N times and vote. If the framework does not beat that at equal cost, it is adding complexity, not accuracy.
Price the gain per correct answer. Voting multiplies tokens by N. On GSM8K, 39 extra GPT-3.5 calls bought 12 points, about 76,000 extra tokens per additional correct answer. Whether that is worth it depends on the cost of a wrong answer, and the comparison with a stronger model at one call: GPT-4 alone scored 0.88 against GPT-3.5's 0.85 with 40 samples.
Voting cannot fix questions the model usually gets wrong. If the most common answer is wrong, more samples make the wrong answer more certain. Voting helps where the model is right somewhat more often than any single wrong answer.
It needs a way to compare answers. Exact match works for numbers and choices; free text needs a similarity measure or a judge, and BLEU between programs is a weak proxy for which one runs correctly.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The method, results by task and model, compatibility, difficulty effects and two misreadings
Free account - 04How it works
Algorithm 1, similarity for each task type, models and settings, and the synthetic difficulty task
Premium - 05The numbers that matter
Table 2 by model and task, combinations, ranking, token use and the difficulty experiments
Premium - 06Try it yourself
Recompute the gains, simulate when voting helps, and price each extra correct answer in tokens
Premium - 07Where it breaks
Independent samples only, cost, similarity for open answers, 2023-era models, and slips
Premium - 08Using it on a deployment
Use voting as the baseline, route only uncertain questions, and compare with a stronger model
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium