GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

GSM8K is the grade-school math benchmark that almost every model report quotes. It is one fixed list of questions, so a score on it says nothing about the same question with different numbers, and its popularity makes it likely that test items leaked into training data.

The authors took 100 questions from the GSM8K test set and rewrote each as a template. In "When Sophie watches her nephew... the bag of building blocks has 31 blocks", Sophie becomes a sampled name, nephew a sampled relative, 31 a variable in a range, and a condition ties the variables together so the answer stays a whole number in a sensible range. Sampling 50 instances of every template gives 50 test sets of 100 questions each. Every set asks the same 100 problems with the same steps.

On these sets, scores varied by several points from set to set, and for most models the score on the 100 originals sat above the middle of the new distribution, which the authors read as a sign of contamination. Changing only names moved scores a little; changing numbers moved them more. Adding one or two clauses to each question lowered accuracy further, and each added clause cost more than the one before. Finally, GSM-NoOp adds a sentence that sounds relevant but changes nothing ("five of them were a bit smaller than average"). Most models turned such clauses into operations, subtracting the five, and accuracy fell by 17.5 to 65.7 points.

Customers ask whether a model "can do math" or "can reason", and usually the evidence is a single public benchmark score. This paper gives you three habits for answering that question honestly.

A score is one draw. Two sets of the same 100 problems with different numbers differed by up to 12 points for Gemma2-9B and about 15 for Phi-3.5-mini. A one- or two-point gap between models on a 100-question test is inside that noise. Ask for the spread, not the point.

Build the test from templates of the customer's own questions. The template method is cheap and portable: take 50 real requests, mark the names, amounts and dates as variables with valid ranges, and generate fresh instances whenever you evaluate. You get a test set that cannot have leaked, and you can measure whether the model is sensitive to the numbers.

Real inputs carry irrelevant clauses. Customer emails, tickets and invoices are full of sentences that sound like they matter and do not. GSM-NoOp is the closest public analogue. If your workflow extracts quantities from messy text, add distractor sentences to the test set deliberately and measure the drop.