Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks

If a model can add, it should be able to add in base 9. The procedure is the same; only the point where a digit carries changes. That is the paper's test, applied to 11 tasks.

Each task gets a default version and one or more counterfactual versions that change a single assumption while keeping the procedure. Arithmetic moves to bases 8, 9, 11 and 16. Code runs in "ThonPy", Python with 1-based indexing. Logic problems keep their structure but make the premises contradict common sense. Coordinates use rotated or swapped axes, drawings must be flipped or rotated, guitar chords use altered tunings, melodies are transposed, chess starts with knights and bishops swapped, and the card game SET inverts the rule for one attribute.

A model could fail a counterfactual task simply by not understanding the instruction, so each one has a counterfactual comprehension check: a much simpler question that only a model that understood the change can answer, such as "what number follows 18 in base 9?"

GPT-4, GPT-3.5, Claude and PaLM-2 were tested in mid-2023, with and without "Let's think step by step." Performance under counterfactual rules was lower on every task, often by tens of points, while comprehension checks were usually high. The authors' reading is that models carry some transferable skill (counterfactual scores were mostly above chance) and a lot of procedure that only works under the familiar conditions.

Every customer has conventions that differ from the internet's defaults, and to a model those conventions are counterfactual worlds.

The customer's rules are the counterfactual. A fiscal year starting in July, week numbers starting on Saturday, an internal SQL dialect, a product code where the second character means region, a spreadsheet indexed from 1. The model has seen the default convention millions of times and the customer's a handful. This paper predicts accuracy will drop even when the model can restate the rule correctly.

Test the rule change directly. Take tasks the model handles under defaults, rewrite them under the customer's convention, and run both. Add a comprehension check, a one-line question about the convention itself, so you can tell "did not understand the rule" from "understood it and still defaulted".

Expect demonstrations to help, not to close the gap. Sixteen worked examples in base 9 raised GPT-4 from 38.6% to 88.4%, still 11.5 points behind base 10. If the convention matters, convert inputs to the default convention in code before the model sees them, or validate outputs against the rule.