Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

Today's reasoning models buy accuracy with tokens: they write a long chain of thought before answering. This paper buys accuracy with depth instead. Its model, released as Huginn, has a transformer block in the middle that it can run as many times as you like before producing each token. More iterations mean more computation per token, with no extra parameters and no extra tokens.

The architecture has three parts. A two-layer prelude turns tokens into an embedding e. A four-layer core block starts from a random state and updates it r times; at every iteration the core sees the current state concatenated with e, which keeps the loop anchored to the input. A two-layer coda turns the final state into next-token probabilities. During training the number of iterations is drawn at random for each step from a heavy-tailed distribution with a mean of 33, and gradients flow only through the last eight iterations to bound memory. The authors trained one 3.5B-parameter model for 800B tokens on up to 4,096 AMD GPUs of the Frontier supercomputer, after two failed attempts.

At test time, accuracy rose with iterations and saturated at different points per task: HellaSwag near 8 iterations, while GSM8K and HumanEval kept using more. At 32 iterations the model scored 38.2% on ARC-Challenge, 34.8/42.1 on GSM8K with chain of thought, and 23.2% on HumanEval, competitive with open models trained on public data except the newest OLMo-2. The same looping structure also gives per-token early exits, KV-cache sharing and speculative decoding without any additional training.

Huginn is a research model, not one to deploy, but it changes how to think about where inference compute goes.

Compute per token can be a dial. With chain of thought, harder requests cost more output tokens. Here, harder requests cost more iterations per token, and the model can decide per token when to stop. If architectures like this reach production, cost and latency per request will vary with difficulty even at a fixed output length, which changes capacity planning and pricing.

Small memory footprint, large compute. At 32 iterations the model reads 3.5B parameters but does the arithmetic of a 52.6B model. Deployments limited by GPU memory and less by compute could fit such a model where a larger dense one would not.

Parameter-matched is not compute-matched. The paper's fixed-depth twin has the same parameters but runs the core once; at 32 iterations the recurrent model spends about 15 times its FLOPs per token. When a vendor says a small model matches a large one, ask what each spends per token.