Training Verifiers to Solve Math Word Problems (GSM8K)

What makes a grade school problem hard

A grade school math word problem is a short paragraph: a situation, a few numbers, and a question whose answer takes two or three steps of arithmetic. No advanced mathematics is involved. What is involved is reading the text, deciding which quantities matter, choosing the order of operations and getting each step right — all at once. That is what makes the task a useful test of whether a language model reasons or only matches surface patterns.

On October 27, 2021, Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen and eight other authors at OpenAI posted a paper on arXiv that turned such questions into a benchmark and proposed a way to score better on it.^[1]

The GSM8K dataset

The paper introduced GSM8K, a dataset of 8.5K high quality, linguistically diverse grade school math word problems.^[1] "Linguistically diverse" means that the problems are not all rewrites of one template: the wording and the way each situation is described vary, so a model cannot do well by memorising a single sentence pattern.

Even the largest models struggled

The authors found that even the largest Transformer models failed to reach high test performance, despite the conceptual simplicity of the problems.^[1] That gap is the starting point of the paper: the problems are easy for people, so what the models lacked was not mathematical knowledge but the ability to turn a paragraph reliably into a sequence of calculations.

Verifiers: generate many, then choose one

The proposed method is to train verifiers. A generating model produces solutions; a verifier is a separately trained model whose job is to look at a question together with a candidate solution and judge whether that solution is correct.^[1] At test time the system generates many candidate solutions for the same problem and selects the one that the verifier ranks highest.^[1]

The idea moves the burden from being right on the first attempt to recognising a right answer among many. The generator does not have to succeed every time, as long as a correct solution appears somewhere among the candidates and the verifier can pick it out. For that reason the method depends on the quality of the verifier's judgement: if it cannot tell wrong solutions from right ones, choosing among candidates does not help.

Results

Verification significantly improved performance on GSM8K.^[1] It also scaled more effectively with increased data than a finetuning baseline — that is, when more data was added, scores rose faster with the verification approach than with straightforward finetuning.^[1]

The experiments used GPT-3 family models as a starting point, focusing mainly on the 175B and 6B sizes.^[2] The authors said the 175B model would likely need at least two more orders of magnitude of training data — roughly a hundred times as much — to reach an 80% solve rate.^[2] The estimate is itself a finding: the gap was not going to close with a modest increase in data.

Why it matters

The paper left two things behind. One is a benchmark: a set of clearly stated, everyday math problems that turns "can a model reason in several steps" into a measurable score.^[1] The other is a methodological point: capability can be improved not only with bigger models and more training data, but also by spending more compute at test time — generating several candidates and having a verifier choose among them.^[1] The estimate of how much data would be needed shows that reaching an 80% solve rate through training data alone would be expensive, while the verification route moved faster for the same added data.^[1][2]