ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

The problem it targets

A benchmark is only useful if it is hard in the right way. ARC-AGI-2, presented in a paper submitted to arXiv on May 17, 2025 by Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers and Henry Pinkard, is a set of tasks built to assess abstract reasoning at higher levels of fluid intelligence in frontier AI systems^[1]. Fluid intelligence here means the ability to work out a novel problem from the few examples in front of you, rather than to apply a pattern already learned during training^[1].

How the tasks work

ARC-AGI-2 keeps the input-output pair task format of its predecessor^[1]. In that format a solver is shown a handful of examples, each pairing an input with its correct output, and must then produce the output for a new input. Nothing in the format states the rule that connects the examples; the rule has to be inferred, and then applied. Keeping the format unchanged matters for practical reasons: researchers who built tooling and intuition around the first version can carry them over without starting again^[1].

What the second version changes

What does change is the task set. ARC-AGI-2 incorporates a newly curated and expanded collection of tasks, designed to provide a more granular signal about abstract reasoning at higher levels of fluid intelligence^[1]. Granularity is the point. A benchmark that every system fails outright stops telling you anything after the first round of results; a benchmark whose scores can move in small steps keeps reporting where the frontier actually sits, and whether a change in a model reflects better reasoning or just better luck.

The paper also reports extensive results from human testing, which serves as a baseline^[1]. That baseline does two jobs. It shows that the tasks are accessible to human intelligence, and it shows that the same tasks remain difficult for current AI systems^[1]. Without a human reference point, a low machine score could mean either that the tasks are genuinely hard or that they are badly posed.

Results from the ARC Prize 2025 competition

The first large public test of the new dataset came through the ARC Prize 2025 global competition, which targeted the newly released ARC-AGI-2 dataset^[2]. It attracted 1,455 teams and 15,154 entries, and the top score reached 24% on the ARC-AGI-2 private evaluation set^[2].

Both halves of that result are informative. The size of the field — well over a thousand teams and more than fifteen thousand submissions — shows that the problem attracts sustained effort rather than a handful of attempts. The top score shows how much of the benchmark remains out of reach: 24% is clearly above zero and clearly short of a solved task. A benchmark in that range is doing its job, because it separates partial progress from the claim that the underlying capability has arrived.

Adoption by frontier labs

In 2025, four frontier AI labs — Anthropic, Google DeepMind, OpenAI and xAI — reported ARC-AGI performance in public model cards^[2]. Model cards are the documents labs publish alongside a model to describe what it can and cannot do; when the same benchmark appears in the documentation of competing labs, it has effectively become a shared yardstick. The ARC Prize 2025 report describes this as establishing ARC-AGI as an industry standard benchmark for AI reasoning^[2].

Toward ARC-AGI-3

The ARC Prize 2025 report also previews the next step, ARC-AGI-3, which introduces interactive reasoning challenges^[2]. Where the current tasks are solved in a single answer, interactive ones require exploration, planning, memory, goal acquisition and alignment capabilities^[2]. In plain terms, a system would have to probe an unfamiliar environment to learn how it behaves, sequence actions toward an objective, carry what it has learned across steps, work out what the objective even is, and stay aligned with the intended goal while doing so. Each of those is a separate demand, and together they describe a harder kind of reasoning than a single inference from examples.

Why it matters

The combination is what gives ARC-AGI-2 its weight: a stated capability target, a human baseline showing the tasks are fair, a public competition that puts a number on the frontier, and adoption by the labs whose systems are being measured^[1][2]. The top score of 24% on the private evaluation set is the concrete reading of where reasoning systems stood against this benchmark^[2].