Measuring Mathematical Problem Solving With the MATH Dataset

The problem

The paper addresses how to measure mathematical problem solving in models. Its title frames the work as measuring mathematical problem solving with the MATH dataset. The authors submitted it to arXiv on March 5, 2021: Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.^[1] For mathematics, a final answer alone does not show whether a model can reason through a problem; a useful evaluation set needs difficult problems and a way to show the steps behind an answer. MATH is presented as that kind of dataset: it is meant to measure problem solving and to supply material for teaching models how to derive answers.^[1]

How MATH is built

MATH contains 12,500 challenging competition mathematics problems.^[1] Each problem has a full step-by-step solution, which can be used to teach models to generate answer derivations and explanations.^[1] The problems are tagged with a difficulty level from 1 to 5 and span seven subjects, including geometry.^[2] In geometry, diagrams can be specified in text with the Asymptote language.^[2] The authors also contributed a large auxiliary pretraining dataset that helps teach models the fundamentals of mathematics.^[1] These design choices serve different purposes: the main problems provide challenging examples with solutions, the difficulty tags allow results to be described at different levels, and the coverage of seven subjects allows results to be examined across areas of mathematics. The auxiliary pretraining data has a different role: it is aimed at the fundamentals a model needs for harder mathematical work.

The role of solutions and tags

The step-by-step solutions mean that a problem can show not only a final answer but the path from conditions to conclusion. That makes the data useful for teaching models to generate derivations and explanations.^[1] The difficulty tags place problems on a 1-to-5 scale, which allows performance to be compared across levels.^[2] The problems cover seven subjects, and geometry problems can carry their diagrams as text written in Asymptote.^[2] These features make MATH more than a list of questions: it is a dataset that can be analyzed by difficulty and subject.

What the experiments showed

Accuracy remained relatively low even with enormous Transformer models.^[1] Models reached up to 15% accuracy on the easiest difficulty level.^[2] The authors found that simply increasing budgets and parameter counts would be impractical for strong mathematical reasoning if scaling trends continued.^[1] The result separates two questions: whether a model can produce an answer, and whether it can produce the steps that lead to an answer. The paper's measurements show that large models still had limited accuracy on MATH, and the authors argue that continuing to enlarge the same kind of model under those trends would not be a practical route to strong mathematical reasoning.^[1]

Why it matters

The paper's contribution is a resource for both measurement and training. The step-by-step solutions support teaching models to produce derivations and explanations, not only final answers.^[1] The auxiliary pretraining dataset addresses the fundamentals of mathematics.^[1] The 1-to-5 difficulty tags and the coverage of seven subjects allow performance to be examined across levels and subjects, including geometry problems whose diagrams are written in Asymptote.^[2] The low accuracy on the easiest level, together with the finding about scaling budgets and parameters, explains why the authors argued that strong mathematical reasoning would be impractical to reach by scaling alone if those trends continued.^[1] MATH therefore serves as a demanding test set and a source of training material for mathematical problem solving. It combines problems, step-by-step solutions, difficulty information, and subject coverage so that models can be measured and taught to generate derivations.