GPQA: A Graduate-Level Google-Proof Q&A Benchmark

A question set that cannot be looked up

GPQA takes its name from the paper's title, Graduate-Level Google-Proof Q&A Benchmark. It is a dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry^[1]. The paper was submitted to arXiv on 20 November 2023 by David Rein, Betty Li Hou, Asa Cooper Stickland and five other authors^[1]. Because every question comes with a fixed set of options, an answer is either the labelled correct one or it is not, and a score can be computed without human grading. What makes the set unusual is the intended difficulty: the questions are meant to be hard for people who know the subject well, and hard to answer by searching.

Why a harder benchmark was needed

A benchmark is a fixed collection of tasks with known answers, used to compare systems on the same footing. Its value depends on being difficult enough to separate them: once strong models answer nearly everything correctly, a high score says little about what a system can and cannot do. Keeping a set hard means asking questions whose answers are not sitting somewhere online in a form that can be retrieved and copied — questions that require graduate-level training in a field to reason through.

How the questions were validated

The authors checked difficulty from two directions. Domain experts — people who hold or are pursuing PhDs in the corresponding field — reach 65% accuracy on the set, and 74% when clear mistakes that the experts themselves identified in retrospect are discounted^[1]. Skilled non-expert validators, given unrestricted access to the web and spending on average more than 30 minutes on a question, reach only 34%^[1]. That gap is the source of the Google-proof label: a patient reader with a browser still gets most questions wrong, so the answer is not simply available to be found and copied.

What AI systems scored

The strongest baseline the authors built, based on GPT-4, reached 39% accuracy^[1]. That is far below the 65% of domain experts and only a few points above the 34% of non-expert validators — at the time, the systems were answering roughly at the level of a careful non-expert rather than a specialist.

Why the authors think it matters

The stated motivation is scalable oversight^[2]. As models take on tasks whose outputs one person cannot fully verify alone, oversight methods need testbeds where a correct answer exists and can be checked, but where the supervising human can neither know it nor look it up. GPQA is meant to be exactly that: difficult for skilled non-experts, difficult for frontier AI systems, yet scoreable against a definite key^[2]. A supervisor can be handed a question and a candidate answer and measured on whether the judgement is right, which is the kind of experiment the authors say the benchmark's difficulty should make realistic^[2].

Limits of the format

The output is a single accuracy number over 448 multiple-choice items, easy to compute and compare, but it records only whether a system picked the right option, not how it reasoned. The Google-proof claim is likewise a statement about the validators and conditions described in the paper: it says that a group of web-equipped non-experts could not answer these questions, not that no question could ever be answered by other means.