OpenAI o3 scores 75.7% (87.5% high-compute) on ARC-AGI-1

Background

ARC-AGI-1 is a benchmark of abstract reasoning tasks. Each task hands the solver a few worked examples and asks for the answer to a new one, and the tasks are built so that memorizing training data does not help: what gets measured is whether a system can work out a rule from a handful of demonstrations. Scores on the benchmark stayed extremely low for a long time, which is why it was watched as a test of whether models can handle genuinely unfamiliar problems rather than reuse patterns seen during training.

The result

On December 20, 2024, the ARC Prize Foundation announced that OpenAI's new o3 system, trained on the ARC-AGI-1 Public Training set, had reached 75.7% on the Semi-Private Evaluation set of 100 private tasks, measured at the public leaderboard's $10k compute limit. A high-compute configuration of the same system scored 87.5% on the same set ^[1].

On the Public Eval set of 400 tasks, o3 scored 82.8% in the high-efficiency configuration and 91.5% in the low-efficiency configuration ^[1].

Cost and compute

The 75.7% figure fell within the budget rules of ARC-AGI-Pub, and so qualified as first place on the public leaderboard. In the low-compute mode it cost roughly $27 per task, against about $5 per task for a human solver ^[1]. The gap widens sharply as compute is added: the high-compute configuration drew 1024 samples per task, roughly 172 times the compute of the 6-sample low-compute configuration ^[1].

What was actually tested

OpenAI shared that the o3 model tested had been trained on 75% of the Public Training set, without giving further details. An update dated April 16, 2025 noted that the officially released o3 is not the same as the version tested in the announcement ^[1]. The numbers therefore describe a particular configuration used for an evaluation, not necessarily the system users could later run.

Reaction

ARC Prize founder Francois Chollet wrote that o3's core mechanism appears to be natural language program search and execution within token space — the model proposes programs in ordinary language and carries them out, rather than retrieving an answer directly. He also predicted that the upcoming ARC-AGI-2 would push o3's score below 30% even at high compute, while a smart human would still score over 95% with no training ^[1]. That forecast places the result in a longer sequence: each jump on a given benchmark has been followed by a harder benchmark meant to keep measuring the thing the old one could no longer separate.

Why it mattered

The announcement set the score against the benchmark's own history. ARC-AGI-1 took four years to go from 0% with GPT-3 in 2020 to 5% with GPT-4o in 2024, which made o3's result a step-function increase in AI capabilities ^[1]. Progress of that shape on a test designed so that memorization does not help raised the question of what the benchmark was still measuring, and the answer offered was a successor version — one on which even a high-compute system was expected to fall well short of an untrained human. For readers trying to follow the field, the episode marks the point where a benchmark that had resisted models for years was suddenly largely solved in its first version, and where the discussion moved to designing the next one.