OpenAI o1 scores 83% on AIME 2024

Background

The American Invitational Mathematics Examination (AIME) is a United States high-school mathematics competition that students reach through earlier qualifying rounds. Every answer is an integer, so marking is mechanical and a result is easy to state: how many of the 15 problems were solved. That makes it a convenient yardstick for multi-step mathematical reasoning, and in September 2024 OpenAI used it as one of the headline measurements in a model announcement.

What was published

On September 12, 2024, OpenAI published a post titled "Learning to Reason with LLMs" introducing o1, described as a new large language model trained with reinforcement learning to perform complex reasoning^[1]. The post compared the new model with GPT-4o on several benchmarks; AIME 2024 was the mathematical one.

The numbers on AIME 2024

On AIME 2024, GPT-4o solved on average 12% of problems (1.8 out of 15). o1 averaged 74% (11.1 out of 15) when it gave a single sample per problem, and 83% (12.5 out of 15) when the answer was chosen by consensus among 64 samples^[1]. An appendix table gives the same comparison in standard notation: pass@1 of 74.4 and cons@64 of 83.3 for o1, against 9.3 and 13.4 for GPT-4o^[1]. The 83% that headlines the result is therefore the consensus figure, not a single-attempt score.

The two labels describe two different procedures. pass@1 grades the model's first and only answer. cons@64 grades an answer picked out of 64 generated samples by agreement among them. The roughly nine-point gap between the two measures what extra computation at answer time buys, as distinct from what the model can do in one shot.

More samples, higher score

The post also reports what happens when sampling is pushed further. Re-ranking 1000 samples with a learned scoring function brought o1 to 93% (13.9 out of 15) on AIME 2024^[1]. A score of 13.9 places it among the top 500 students nationally and above the cutoff for the United States of America Mathematical Olympiad^[1]. Taken together, the three figures — 74%, 83% and 93% — trace a curve in which accuracy rises as more samples are generated and more effort goes into choosing among them.

Competitive programming

The announcement measured a second kind of reasoning on Codeforces, a platform for competitive programming contests in which participants carry an Elo rating. On Codeforces questions, o1 reached an Elo of 1673, placing it in the 89th percentile of participants; o1-preview reached 1258, the 62nd percentile, and GPT-4o reached 808, the 11th percentile^[1]. Competitive programming problems reward the same combination of reading a precise statement, finding a method and carrying it through without slips, so the result was presented alongside the mathematics scores rather than as a separate claim.

Why the result mattered

Two things in the report stand out. The first is the size of the single-sample gap: 74% against 12% means the improvement is not merely an artefact of drawing many samples and picking a good one, since the model's first answer alone already clears most of the distance. The second is how visibly the score depends on the sampling budget. Going from one sample to a consensus of 64, and then to re-ranking 1000, moves the same model from 74% to 83% to 93%, which makes the inference procedure part of the result rather than a detail. Reading such figures requires knowing which protocol produced them; a single percentage point without that context can be misleading.

It is also worth noting where the numbers come from. These are results published by the developer of the model in its own announcement, not an independent third-party evaluation, and the comparison is against the same company's earlier GPT-4o. The AIME and Codeforces figures are best read as a description of what was reported on those two benchmarks on that date, and as a template for how later reasoning models would be presented: a hard exam, a single-sample score, and a score under a larger sampling budget side by side.