Think before you speak: Training Language Models With Pause Tokens

To produce its next token, a decoder-only transformer computes one vector per preceding token in every layer. If the prompt has K tokens, each layer gets K positions of computation before the answer, no more. The authors ask what happens if the model may compute over K + M positions, by appending M dummy tokens and waiting to read the answer until the last one.

They use one new learned token, <pause>, outside the normal vocabulary. In pretraining, they insert pauses at random positions, about 10% of each 2,048-token sequence, and never ask the model to predict a pause. In finetuning and at inference, they append 10 or 50 pauses to the prompt and ignore the model's outputs until the last pause, then read the answer. The only new parameters are the pause embedding's 1,024 numbers.

With a 1B-parameter model pretrained on 200B tokens of C4, pauses in both pretraining and finetuning beat standard training on 8 of 9 downstream tasks. SQuAD exact match rose from 36.4 to 55.9, CommonSenseQA from 26.9 to 34.8, and GSM8k from 7.5% to 8.5%. Two conditions mattered. Adding pauses only at finetuning, to a normally pretrained model, gave small and inconsistent gains and sometimes hurt. And the pause-trained model needs its pauses: with zero pauses at inference its performance collapsed, though as few as two brought it back.

Pause tokens are a training technique, not something you can add to a hosted model, but the paper settles a question that comes up in prompt engineering and explains why some tricks do not work.

Padding a prompt with dots does not buy computation. The authors confirm that appending periods to a normally trained model gave no gains, as earlier work found. Extra positions help only when a model has been trained to use them. If someone proposes "give the model room to think" by padding prompts, the evidence says it will not help an off-the-shelf model.

Computation can come from width as well as length. Chain of thought adds sequential steps. Pauses add parallel positions per layer without adding output text, which suits tasks like reading comprehension where the model must look in many places at once. The SQuAD gain, far larger than GSM8k's, fits that.

Training choices show up at inference. A model trained with pauses may depend on them; this one failed without any. When a vendor's model uses special tokens or formats learned in training, deviating from them at inference can be costly, so the serving template is part of the model.