Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

The earlier STaR paper taught a model to reason by keeping its own rationales whenever they reached a labeled answer. That needs a question-answer dataset. Quiet-STaR, from the same group, drops the labels. Any text has reasoning implied between its lines: the step skipped in a proof, the reason a sentence follows the last. If the model writes a short thought before predicting the next token, the real next tokens act as the answer key.

The method has three steps, which the authors call think, talk and learn. Think: at every position in a training sequence, the model generates a thought of 8 to 24 tokens between learned start-of-thought and end-of-thought tokens. A special attention mask lets all positions generate their thoughts in parallel, one token per forward pass. Talk: a small mixing head looks at the hidden states with and without the thought and decides how much the after-thought prediction should count. Early in training, thoughts are mostly noise, and the mixing head keeps them from hurting the model. Learn: each thought is scored by how much it raises the log-probability of the true next several tokens, relative to the other thoughts sampled at that position, and the better-than-average thoughts are reinforced with REINFORCE.

Applied as continued pretraining to Mistral 7B on OpenWebMath, a crawl of math-heavy web pages, it raised zero-shot GSM8K from 5.9% to 10.9% and CommonsenseQA from 36.3% to 47.2%, with no training on either task. Training on C4 helped less. On average, next-token prediction barely improved; the gains were concentrated on hard tokens. The cost is large, because every token now comes with a generated thought.

Quiet-STaR is a research precursor to models that think before they answer, and it makes the price of that visible in a way product demos do not.

Thinking is paid for in tokens. Each output token here comes with a hidden thought of up to 24 tokens plus two marker tokens, so decoding takes roughly 10 to 27 times the steps of plain generation by our count. When a customer compares a reasoning model with a standard one, the latency and cost questions are the same: how many hidden tokens per visible token, on which inputs.

Unlabeled text can teach some reasoning, slowly. Gains came from raw web text with no curated answers, which is the appeal. But a 7B model still scored 10.9% on GSM8K zero-shot. This is a signal that self-supervised reasoning helps, not a way to get a strong reasoner cheaply.

Hidden thoughts are not explanations. The authors state that nothing guarantees the thoughts reflect what the model is computing, and nothing filters harmful reasoning the model finds useful. Treat them as a mechanism for better prediction, not as an audit trail.