Humanity's Last Exam (HLE)

Humanity's Last Exam (HLE) is a benchmark paper by Long Phan, Alice Gatti, Ziwen Han and a large list of co-authors, submitted to arXiv on January 24, 2025 ^[1].

Why another benchmark

A benchmark is only useful while models still fail part of it. By the time this paper appeared, large language models had largely saturated the tests that had been used to track them: on popular benchmarks such as MMLU — a widely used multiple-choice test of academic knowledge — models were reaching over 90% accuracy ^[1]. Once scores crowd against the ceiling, a benchmark stops separating one system from another, and it becomes hard to make any informed statement about what the state of the art can and cannot do ^[1]. HLE was proposed as an answer to that problem.

What the authors built

HLE is a multi-modal benchmark placed at the frontier of human knowledge, with broad coverage across fields ^[1]. The authors describe it as designed to be the final closed-ended academic benchmark of its kind ^[1]. "Closed-ended" means every question has a definite answer that can be checked mechanically, as opposed to open tasks where judging quality requires argument.

The benchmark contains 2,500 questions across dozens of subjects, including mathematics, the humanities and the natural sciences ^[1]. The questions were written by subject-matter experts around the world ^[1]. Two formats are used — multiple-choice and short-answer — and both are suitable for automated grading ^[1]. That choice matters for a benchmark that has to be run again and again against many models without a human reading every response.

Quality control rests on two properties that pull in opposite directions. On one side, each question has a known solution that is unambiguous and easily verifiable, so a grader can decide without dispute whether an answer is correct ^[1]. On the other, the questions were constructed so that they cannot be quickly answered through internet retrieval ^[1]. Together these rule out two common ways a benchmark loses its value: answers that are contested, and answers that a model can simply look up.

What the results showed

Evaluated on HLE, state-of-the-art language models demonstrated low accuracy and poor calibration ^[1]. Low accuracy means they got relatively few questions right. Calibration is a separate measure: it asks whether a model's stated confidence matches how often it is actually correct. A model that answers "I am certain" and is right most of the time is well calibrated; one that is equally confident when wrong is not. HLE exposed weakness on both counts ^[1].

Why it matters

The result is a measuring instrument that is hard to game and hard to saturate. Because questions are unambiguous, verifiable and not solvable by quick lookup, a score on HLE says something about knowledge and reasoning rather than about memorised test formats or search skills ^[1]. Because the questions come from experts across many disciplines, the benchmark samples a wide range of expert-level knowledge instead of one narrow skill ^[1].

HLE also marks a deliberate end point. The authors frame it as the final closed-ended academic benchmark of its kind, implying that further progress in this format would add little, and that future evaluation would need to move toward tasks that cannot be scored as a single correct answer ^[1]. Read that way, the paper is as much a statement about the limits of multiple-choice benchmarking as it is a new dataset.

In short

HLE takes the design of an academic exam — many subjects, expert-written, automatically gradable, one verifiable answer per question — and pushes it to a difficulty where frontier models score poorly and misjudge their own reliability ^[1]. That combination is what makes it useful as a yardstick at a moment when easier yardsticks had stopped measuring anything ^[1].