Test-time compute

What the term covers

A language model's ability comes from two separate pools of computation. One is spent during training, when the weights are set and then frozen. The other is spent at the moment of answering a prompt: extra sampling, extra search, extra checking against a verifier. That second pool is test-time compute^[1]. The distinction matters because the two pools are paid for in the same currency — floating-point operations — but are decided at different times. One budget is fixed before training begins; the other can be decided afresh on each question.

The question the term raises is what the second pool actually buys. In August 2024, Snell et al. published a study of test-time scaling asking how much a model can improve on a challenging prompt when it is allowed a fixed but non-trivial amount of inference-time compute^[1]. The framing is deliberately budget-like: hold the extra computation fixed, and see how much of it is well spent.

Two ways to spend the budget

The paper analyses two mechanisms^[1].

The first is search against dense, process-based verifier reward models^[1]. In this setup the model does not commit to a single answer. It produces candidates, and a separate verifier scores them. Process-based means the verifier grades the intermediate steps of a solution rather than only its final answer; dense means it returns a graded score that can steer a search step by step, rather than a single pass-or-fail signal. Search then uses those scores to decide where a line of reasoning is worth continuing.

The second mechanism is adaptively updating the model's distribution over a response, given the prompt^[1]. Rather than treating the model's first-pass distribution over answers as fixed, this approach re-shapes that distribution in light of the question being asked, so that probability mass moves toward the kinds of responses the prompt calls for.

Both convert extra inference computation into a better answer, but they consume the budget differently: one spends it on generating and grading many attempts, the other on refining how the model responds in the first place.

Difficulty decides which one pays

The authors' central empirical observation is that the effectiveness of each approach varies critically with the difficulty of the prompt^[1]. Which mechanism is the better buy is not a property of the mechanism on its own; it depends on the problem in front of the model.

That observation motivated a compute-optimal strategy: rather than applying one fixed recipe to every prompt, allocate test-time compute adaptively, prompt by prompt^[1].

What the strategy delivered

Two results stand out.

Against a best-of-N baseline — the straightforward recipe of generating N candidates and keeping the best — the compute-optimal strategy improved the efficiency of test-time compute scaling by more than 4x^[1]. Comparable quality for a fraction of the inference budget, or more quality for the same budget.

More striking is the FLOPs-matched comparison. When total computation was held equal, test-time compute let a smaller base model outperform a model 14 times larger^[1]. The comparison is conditional: it holds on problems where the smaller model already had somewhat non-trivial success rates^[1]. Extra inference computation amplifies a capability present in a weak form; the study does not claim it conjures one that is absent.

Why the result travels beyond one paper

The authors frame the finding as relevant to how future pretraining budgets should be traded off against inference budgets^[1]. If part of the compute that would have gone into a larger pretraining run can instead be spent at answer time, then more capability no longer automatically means a bigger training run. The two budgets become things to be balanced against each other. This is what the paper's own title points at: scaling test-time compute optimally can be more effective than scaling model parameters.

The conclusions come with clear boundaries. The FLOPs-matched win is established on problems where the smaller model already succeeds at least occasionally^[1], and the strategy works by allocating effort per prompt, so its payoff depends on that allocation being made well^[1]. The paper offers a map of when inference compute is spent well, not an unconditional shortcut.