On the Measure of Intelligence (ARC-AGI)

The problem the paper addresses

On 5 November 2019, Francois Chollet submitted a paper titled *On the Measure of Intelligence* to arXiv^[1]. Its subject is not a new model but the yardstick used to judge models. The paper argues that making deliberate progress towards more intelligent and more human-like artificial systems requires intelligence to be defined and evaluated in a way that allows comparisons between two systems, and between a system and a human^[1].

The reasoning behind that demand is straightforward. A score on a benchmark is only meaningful relative to the tasks, the scoring rule and the assumptions built into them. If two systems are measured on different tasks, or if a task is scored in a way that has no counterpart in human performance, the resulting numbers cannot say which system is closer to human-like intelligence. A common definition, in other words, is what turns a collection of results into a direction of travel.

A definition built on Algorithmic Information Theory

The paper does not stop at criticism; it proposes a formal definition of intelligence grounded in Algorithmic Information Theory^[3]. In this framing, intelligence is described as skill-acquisition efficiency rather than as skill itself: what matters is how much skill a system can acquire for a given amount of prior knowledge and experience, not how well it performs on any single task it was built for. The definition singles out four things that have to be pinned down before any comparison can be made: the scope of tasks under consideration, the generalisation difficulty involved, the priors the system starts with, and the experience it is allowed to draw on^[3].

These four factors are what make the definition usable in practice. A system that performs well on tasks close to what it has already seen, or that has been given a great deal of relevant built-in structure, is not being credited with the same achievement as one that reaches the same performance with less to start from. Making scope, generalisation difficulty, priors and experience explicit is what allows two systems, or a system and a person, to be placed on the same scale.

The Abstraction and Reasoning Corpus

To give the definition something to measure, the paper introduced a benchmark: the Abstraction and Reasoning Corpus, later renamed ARC-AGI to avoid a naming collision with other AI benchmarks^[2]. The original dataset, now referred to as ARC-AGI-1, is a set of tasks. Each task consists of demonstration pairs and test inputs, all of them grids of up to 30 by 30 cells in which each cell takes one of ten distinct values^[2]. A solver is shown the demonstration pairs, has to infer the underlying rule that maps one grid to another, and then applies that rule to each test input to construct the corresponding output grid; two attempts are allowed per test input^[2].

What the tasks leave out is as deliberate as what they contain. Solving them does not require specialised world knowledge or language; the benchmark assumes only Core Knowledge concepts such as objectness, basic topology and elementary integer arithmetic^[2]. A task therefore cannot be answered by recalling a fact, translating a sentence or retrieving a remembered example. The solver has to notice which parts of a grid behave as objects, see how those objects relate to one another, and work out the transformation from a handful of examples — the kind of abstraction that human beings perform with very little instruction.

Why it matters

The paper's contribution is conceptual and methodological rather than a report of a system: it offers a definition of intelligence that can be applied to machines and humans alike, and a benchmark designed around the priors that definition treats as minimal^[1][2][3]. Because the tasks assume only Core Knowledge, performance on them is not a proxy for the amount of domain data a system has absorbed, which is what makes the benchmark a candidate for comparing systems with each other and with people.

The later history of the name reflects how the benchmark was received: the corpus came to be called ARC-AGI, a change made to avoid confusion with other AI benchmarks^[2]. The original task set is now known as ARC-AGI-1, which situates it as the first in a line rather than a one-off dataset^[2].