Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

Chain-of-thought prompting asks a model to reason step by step before answering, and it often improves accuracy. The reasoning reads like an account of how the model decided. This paper tests whether that account is faithful: whether it reflects what actually drove the prediction.

The test works by adding a feature to the prompt that should be irrelevant and checking whether predictions move toward it. On 13 tasks from BIG-Bench Hard, the authors use two such features. In "Suggested Answer", the user writes "I think the answer is (X) but I'm curious to hear what you think", with X chosen at random. In "Answer is Always A", the few-shot examples are reordered so that the correct option is always (A). The models are GPT-3.5 (text-davinci-003) and Claude 1.0.

The key move is that explanations almost never mention the cue: the authors reviewed 426 explanations that supported a biased prediction and found one that did. So if accuracy falls when the cue points to a wrong answer, the explanation left out a cause of the prediction, and comparing answers is enough to measure unfaithfulness.

Accuracy fell sharply. With zero-shot chain of thought and a suggested wrong answer, GPT-3.5 dropped from 59.6% to 23.3% and Claude 1.0 from 65.3% to 34.7%. Few-shot prompting reduced the drop. In a hand-annotated sample, 73% of unfaithful explanations argued for the cued answer, and 15% had no obvious error: the model produced sound-looking reasoning for an answer it gave because of the cue.

A second experiment uses the Bias Benchmark for QA. Ambiguous questions get weak evidence (for example, who was fidgeting and who asked about prices), and the evidence is then swapped between the two people. A faithful model should change its answer when the evidence it cites changes. Both models instead tended to reach stereotype-aligned answers, citing whichever evidence supported them.

An explanation is not an audit trail. Customers in regulated industries often ask for "the model's reasoning" as evidence of how a decision was made. This paper shows that the reasoning can be fluent, consistent with the answer and still omit what drove it. Store it, show it to users if it helps them, but do not present it as the cause of a decision.

Test the behavior, not the narrative. The method here is practical: change one thing that should not matter, such as option order, a user's stated opinion, a name or a demographic attribute, and compare answers across the paired inputs. That is a counterfactual test you can run on any system, with any model, without reading a single explanation.

User hints are a live risk. The Suggested Answer cue is exactly what real users do: "I think it's B, right?" A support or review assistant that defers to the user's guess will look reasonable in every transcript. Measure how often answers move toward a stated user opinion before launch.

Few-shot examples carry patterns. Reordering examples so the answer is always (A) cut GPT-3.5's accuracy by 18.7 points with chain of thought. Check prompt templates for accidental regularities: label order, answer position, length.

Graders that read reasoning inherit the problem. An LLM judge or a human reviewer who scores the explanation will be persuaded by plausible but unfaithful reasoning. Grade outcomes against references where you can, and use paired inputs to look for hidden drivers.