Let's Think Dot by Dot: Hidden Computation in Transformer Language Models

When a model writes a chain of thought, two things change: the text contains useful intermediate steps, and the model gets more forward passes before it answers. This paper separates the two. It replaces the chain of thought with filler, a string of dots, which carries no information. If accuracy still rises, the gain came from the extra computation, and that computation is invisible to anyone reading the tokens.

Large commercial models showed no benefit from filler in earlier tests on QA and math benchmarks. The authors argue those tasks were the wrong test. Theory says a transformer answering in one pass can solve only fairly parallel problems, and filler tokens add parallel width rather than sequential steps. So they built tasks that need parallel width. In 3SUM, the model sees a list of small vectors and must say whether any three sum to zero modulo 10. Its chain of thought lists every pairwise sum, which is exactly the kind of work each dot's hidden state can do in parallel. In 2SUM-Transform, the inputs are scrambled by a transformation revealed only at the end, so no work can be done until the last token.

A small 4-layer, 34M-parameter Llama trained on a mix of chain-of-thought and filler sequences solved length-12 3SUM perfectly from filler alone, while models without intermediate tokens stalled at 66%. On 2SUM-Transform, filler scored 93.6% against 95.1% for chain of thought and 78.7% for an immediate answer. The model learned to use filler only when trained with chain-of-thought examples whose steps were parallel; filler-only training or serial chain of thought did not teach it.

This is a small-model, synthetic-task paper, but it sharpens a question that comes up in every regulated deployment: does the reasoning a model shows explain how it reached the answer?

Visible reasoning is not a full record. The paper shows that intermediate tokens can carry computation unrelated to what they say. When a customer plans to audit decisions by reading a model's chain of thought, explain that the text is evidence about the reasoning, not a transcript of it, and keep independent checks on the decisions themselves.

Monitoring outputs has a blind spot. The authors point out that most alignment and safety methods judge output tokens. Computation hidden in filler cannot be judged that way. Output monitoring is still useful; it is not sufficient on its own for high-stakes systems.

Current models do not seem to do this, and that can change. Earlier tests found no filler benefit in Claude 2 or GPT-3.5 on QA and math, and this paper needed special training to get it in a small model. The result says the capacity exists in the architecture, not that deployed models use it.