GAIA: a benchmark for General AI Assistants

The problem

Benchmarks shape what AI systems are optimized to do. In recent years, many evaluations have asked models to perform tasks that are hard for people, such as professional-level questions in law or chemistry. GAIA starts from a different direction: it asks whether an AI assistant can handle questions that an ordinary person can answer, but that require several basic abilities at once. The authors—Gregoire Mialon, Clementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom—submitted the paper to arXiv on November 21, 2023.^[1]

How it works

GAIA proposes real-world questions that require reasoning, multi-modality handling, web browsing, and general tool-use proficiency.^[1] These are not necessarily specialized professional skills. A question can require a system to read information in different formats, follow a chain of reasoning, check a source online, or use a tool to complete a step. The benchmark is designed around the combination of abilities that a general assistant needs, rather than a single narrow task.^[1]

Using GAIA's methodology, the authors devised 466 questions and their answers. The questions are released, while answers to 300 of them are retained to power a public leaderboard.^[1] This lets researchers see the questions and work on them, while keeping a scored subset for comparing systems. A leaderboard gives the field a common reference point: systems are judged on the same held-out answers, and progress can be tracked over time.

The results

The paper reports a large gap. Human respondents obtain 92% on GAIA, while GPT-4 equipped with plugins obtains 15%.^[1] The questions are conceptually simple for humans yet challenging for most advanced AIs.^[1] This is the central result: a benchmark that is not built around expert-level difficulty can still expose weaknesses in AI assistants.

Why it matters

The disparity contrasts with the recent trend of large language models outperforming humans on tasks requiring professional skills in areas such as law or chemistry.^[1] That trend can suggest that AI is closing in on general human ability. GAIA argues for a different test: robustness on questions that an average human can handle. The authors posit that the advent of artificial general intelligence (AGI) hinges on a system's capability to exhibit robustness similar to the average human on such questions, departing from the trend of targeting tasks that are ever more difficult for humans.^[1]

What the benchmark measures

GAIA is not only a set of trivia questions. It measures whether a system can combine reasoning, multi-modality handling, web browsing, and tool use in a realistic workflow.^[1] For a general assistant, those abilities matter because users hand over tasks that span steps and sources. The benchmark has 466 questions, with 300 answers held back for the leaderboard.^[1] Its focus is not to cover every professional domain, but to make average human performance a target for general AI assistants.