Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Model Parameters

A model can use extra computation at inference in several ways. It can sample many answers and pick one with a verifier (best-of-N). It can search step by step, scoring partial solutions with a process reward model (PRM) that rates each step. Or it can revise its own answer repeatedly. Earlier papers disagreed about whether any of this helps on hard reasoning tasks. This paper, by Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar at UC Berkeley and Google DeepMind, asked two cleaner questions: given a fixed inference budget, how should you spend it, and when is it better than using a bigger model?

They worked on the MATH benchmark of competition problems with PaLM 2-S*, fine-tuned for two jobs: a verifier that scores each solution step, trained without human labels using Monte Carlo rollouts, and a revision model that sees its previous wrong answers and tries again.

The key move was measuring difficulty from the model's own point of view: sample 2,048 answers per question and bin questions into five levels by how often the model is right. Then pick, for each difficulty level and budget, the best way to spend compute. Easy questions did best with sequential revisions and, at high budgets, best-of-N; harder questions did best with beam search against the verifier and a mix of parallel and sequential sampling. The hardest questions barely improved with any method.

Choosing strategy by difficulty, which they call "compute-optimal" scaling, matched best-of-N with up to 4× less compute. And in a comparison at equal total FLOPs, a small model with optimally spent test-time compute beat a model 14× larger on easy and medium questions, especially when inference traffic is light relative to pretraining. On the hardest questions, or with heavy inference traffic, the bigger model was the better buy.

Customers regularly face the trade this paper measures: pay for a larger model on every request, or run a smaller one with more effort on each request, through sampling, voting, verification or reasoning. The answer depends on the workload, and this paper explains how.

Spend compute where it changes the answer. Easy requests are solved on the first try, and hopeless ones stay unsolved however long the model thinks. Extra compute pays off in the middle. A router that estimates difficulty and allocates effort accordingly beats a fixed setting.

Test-time compute does not replace capability. On questions beyond the small model's reach, no amount of sampling or revising caught up with the larger model. If a customer's hardest tasks are where the value is, size up the model.

Traffic volume changes the economics. Pretraining cost is paid once; inference is paid on every request. With heavy traffic, spending more per request costs more in total than a bigger model; with light traffic or offline batch work, extra inference compute is cheap.

Verifiers matter as much as generators. Search only helps if the scoring is good. The authors found a public verifier dataset, made with a different model, easy to exploit, and had to train their own.

Self-revision needs a selection step. A naive revise-again loop turned 38% of correct answers wrong. Keep all candidates and choose with a verifier or a vote.