IN THIS BREAKDOWN · 10 SECTIONS
You can read 20% of this breakdown.
The paper in one minute
A language model doing chain of thought writes each reasoning step in words, then reads its own words back. Every step passes through a bottleneck: the model's rich internal state is collapsed into one chosen token. Most of those tokens exist for fluency, and committing to one token means committing to one reasoning path.
Coconut (Chain of Continuous Thought) removes the bottleneck for part of the reasoning. After the question, a begin-thought marker switches the model into latent mode. In latent mode, the last hidden state at each position is fed straight back in as the next input embedding, with no token chosen. After a fixed number of these continuous thoughts, an end-thought marker switches back to ordinary text for the rest of the answer. Because everything stays continuous, the whole chain can be trained with gradient descent.
The model does not discover useful continuous thoughts from questions and answers alone; trained that way, it did no better than answering directly. Instead, it learns in stages from written reasoning: stage 0 trains on full chain of thought, stage 1 replaces the first written step with continuous thoughts, stage 2 the first two, and so on. With GPT-2, Coconut scored 34.1% on GSM8k with 8.2 generated tokens, against 42.9% for chain of thought with 25 tokens. On ProsQA, a new graph-search task, it reached 97.0% against chain of thought's 77.5%. Probing the hidden states suggests they hold several possible next steps at once, so the model can explore alternatives breadth-first before choosing.
Research notes: the problem, approach and results
ProsQA accuracy for Coconut with 14.2 generated tokens, against 77.5% with 49.4 for chain of thought (Table 1)
GSM8k accuracy for Coconut, between no chain of thought (16.5%) and chain of thought (42.9%) (Table 1)
fewer steps than chain of thought on GSM8k (8.2 against 25.0), and about 2.9x less time per problem (Table 1, Appendix B)
Chain of thought makes a model reason in words, which spends most tokens on fluency and forces it to commit to one next step at a time. On problems that need search, a model that commits early to a wrong branch tends to invent facts or reach the wrong target.
Let the model reason in its own hidden states. In latent mode, the last hidden state is fed back as the next input embedding instead of being decoded to a token. Train it with a curriculum that replaces the first k written reasoning steps with continuous thoughts, stage by stage.
With GPT-2, Coconut reached 34.1% on GSM8k with 8.2 tokens, against 42.9% with 25 tokens for chain of thought and 16.5% without it. On ProsQA it scored 97.0% against 77.5% for chain of thought, though a curriculum with no continuous thoughts reached 95.5%. Probing suggests the hidden states hold several candidate next steps at once.
Why it matters to an FDE
Coconut is research on small models, not something to deploy, but it frames two questions customers ask about reasoning models.
Written reasoning is expensive, and not all of it is reasoning. Coconut used a third of chain of thought's tokens on GSM8k and a tenth on ProntoQA, and clock time fell roughly in proportion. When a customer pays for long visible reasoning, the question this paper raises is how much of it the model needs to emit, and the answer here is "less than chain of thought writes, at some cost in accuracy on math."
Hidden reasoning cannot be read. Continuous thoughts are vectors, not text. Researchers could only inspect them by decoding them as if they were tokens. For workflows where reasoning must be reviewed or logged, latent reasoning removes the record; that is a governance question to raise before any such model is adopted.
Read the ablation, not the headline. On ProsQA, a curriculum with no continuous thoughts at all scored 95.5% and one with pause tokens 96.6%, close to Coconut's 97.0%. The gain over chain of thought there comes mostly from the training schedule. When a vendor credits a new mechanism, ask for the ablation that removes the mechanism and keeps everything else.
Understand the mechanism. Try it on a real problem.
Unlock this breakdown and 290 more papers.
Premium opens every paper in full, plus every answer, concept, course and guide on the site. One payment for six months of access.
6 months · One payment · No auto-renewal
- ◇1 architecture diagram
- </>1 runnable code experiment
- ▦2 results tables
- ?5 interview questions with answers
- 03Key takeaways
How latent mode works, why the curriculum matters, the search interpretation and two misreadings
Free account - 04How it works
Latent mode, the curriculum, training costs, inference, and the ProsQA probing setup
Premium - 05The numbers that matter
Table 1 across three tasks, the ablations, clock time, larger models and the search evidence
Premium - 06Try it yourself
Read Table 1 with its spreads, check clock time per step, and test the search argument on a toy
Premium - 07Where it breaks
Small models, behind chain of thought on math, sequential training, opacity and slips
Premium - 08Using it on a deployment
What to ask about latent reasoning, how to test efficiency claims, and audit implications
Premium - 09Interview angles
Five questions this paper prepares you for, with answer hints
Premium - 10Read next
Where to go from here on this site
Premium