IN THIS BREAKDOWN · 10 SECTIONS
You can read 18% of this breakdown.
The paper in one minute
"Alice has N brothers and she also has M sisters. How many sisters does Alice's brother have?" The answer is M + 1: Alice's sisters, plus Alice. An adult solves it in seconds.
The authors pose this problem (AIW, for Alice in Wonderland) in four variations that change only N and M: 3 brothers and 6 sisters, 4 brothers and 2 sisters, 1 brother and 4 sisters, 4 brothers and 1 sister. Each is asked with three prompt types (a plain question, a "think carefully" prompt and a short-answer prompt) at least 30 times, and the correct response rate is the share of trials with the right number.
Most models in mid-2024 scored close to zero, including several with high benchmark scores. GPT-4o averaged 0.649, Claude 3 Opus 0.431 and Llama-2 70B Chat 0.3. The averages hide a second failure: the better models' rates swung between variations, near 1 on one and near 0 on another, although the problem was identical.
Control problems ("how many siblings does Alice have?", "how many brothers does Alice's sister have?") were solved by models that failed AIW, so the cause is not parsing or arithmetic. Harder versions extended the result to newer models: o1-preview solved one consistently, then wavered on another, and o3-mini (medium) went from above 0.9 on one variation to below 0.1 on another. Wrong answers often came with confident, plausible explanations, and models rarely fixed them when asked to check.
Research notes: the problem, approach and results
GPT-4o's average correct rate on AIW across variations and prompt types; Claude 3 Opus 0.431 (Section 3.1)
o3-mini (medium) on AIW Circles Colleagues: above 0.9 on one variation, below 0.1 on another (Figure 30)
AIW variations where adding the two numbers also gives the right answer, so a shortcut scores (our check)
Benchmarks such as MMLU and GSM8K report high scores for large models, yet users find simple questions they get wrong. It was unclear whether such failures are rare quirks or a sign of weak generalization that benchmarks do not measure.
Pose one short common-sense problem with a fixed structure (Alice has N brothers and M sisters; how many sisters does her brother have?) and vary only N and M. Run 30 to 60 trials per variation and prompt type, add control problems that test the sub-skills, and extend to harder versions.
Most models scored near zero, GPT-4o and Claude 3 Opus averaged 0.649 and 0.431, and better models swung from near 1 to near 0 between variations. Control problems were solved, so parsing and arithmetic were not the issue. Later reasoning models improved but still swung across variations of harder versions.
Why it matters to an FDE
Averages hide the failure mode customers will hit. A model that scores 0.65 on average may be near 1 on some phrasings of a request and near 0 on others. If production traffic concentrates on one phrasing, the average is meaningless. Report per-variation results.
Small, structure-preserving variations are a cheap probe. One problem, four instantiations, 60 trials each: a few hundred calls show whether a model's skill on a task transfers across inputs that should not matter. The same pattern works for any customer workflow.
Confident wrong answers need outside checks. The paper documents wrong answers wrapped in persuasive explanations, and models that did not revise when asked. Where users cannot verify the answer themselves, a reviewer, a rule or a calculation has to.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
The problem and its variations, prompt types, control problems, newer models and two misreadings
Free account - 04How it works
Problem versions, prompt types, trials and scoring, controls, model selection and later tests
Premium - 05The numbers that matter
Average rates, the per-variation swings, controls, newer models and fine-tuning
Premium - 06Try it yourself
Find shortcut answers per variation, size the sampling error, and compute guessing floors
Premium - 07Where it breaks
One problem family, coincident shortcuts, an assumption about siblings, moving models, and wording
Premium - 08Using it on a deployment
Build structure-preserving variations, report per-variation rates, and check for shortcuts
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium