FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI

A benchmark built around hard problems

A benchmark is a fixed set of problems with checkable answers that lets researchers compare different AI systems on the same task. What a benchmark can actually tell you depends on how hard its problems are, and FrontierMath was designed around a deliberately high bar. It is a benchmark of hundreds of original, exceptionally challenging mathematics problems, crafted and vetted by expert mathematicians, introduced by Epoch AI with an announcement first published on November 8, 2024^[1][2].

How the problems were made

The problems are new and unpublished rather than drawn from textbooks or competition archives. FrontierMath pairs these new problems with automated verification, so that answers can be checked by machine, while keeping the risk of data contamination low^[1]. Contamination is the problem that arises when benchmark questions end up in the text a model is trained on, which can make a model look better than it is.

The benchmark was developed through collaboration with over 60 mathematicians from leading institutions. Among the contributors were professors, writers of International Mathematical Olympiad questions, and Fields medalists^[2]. The problems are designed to be "guessproof", with less than a 1% chance of guessing correctly^[2]. The paper acknowledges OpenAI for support in creating the benchmark, and thanks Terence Tao, Timothy Gowers and Richard Borcherds for interviews; Terence Tao also contributed several problems^[1].

What the problems cover

The questions span most major branches of modern mathematics. They range from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory^[1].

Difficulty was set with working mathematicians in mind: solving a typical problem takes a researcher in the relevant branch of mathematics multiple hours of effort, and the upper-end questions can take multiple days^[1]. The benchmark is therefore aimed at research-level reasoning rather than at school or olympiad exercises, even though some of its contributors write olympiad questions.

Results at release

At release, the strongest current AI models solved under 2% of the problems^[1]. That result was presented as evidence of a wide gap between what AI systems can do and the capability of the mathematical community^[1].

Why the design matters for reading the score

A low score on a hard test is only meaningful if the test cannot be passed by other means. Two design choices in FrontierMath address this directly. Because the problems were newly written and not previously published, a model is unlikely to have seen them during training, which lowers the chance that a score reflects memorization rather than reasoning^[1]. Because the problems are built to be guessproof, with a chance of a correct guess below 1%, a high score cannot come from lucky answers either^[2]. Automated verification then keeps the checking of answers consistent as models are evaluated^[1].

Together these choices mean the reported figure of under 2% describes performance on problems that take a specialist hours or days, on material that was not public beforehand. The benchmark thus records a specific point of comparison between AI systems and working mathematicians, and gives later evaluations a fixed set of problems against which progress on advanced mathematical reasoning can be measured^[1].