Measuring Massive Multitask Language Understanding (MMLU)

Most tests for language models ask for one kind of skill: answer questions about a passage, resolve a pronoun, finish a sentence. The authors of this paper wanted something broader — a single test that asks whether one model holds knowledge across many different fields and can use it to reason. Their answer was MMLU, a benchmark assembled from questions spanning dozens of subjects and scored in one comparable way. The paper was submitted to arXiv on September 7, 2020 by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt ^[1].

What the test contains

MMLU covers 57 tasks, among them elementary mathematics, US history, computer science and law ^[1]. The mix is deliberate. It runs from school-level arithmetic to professional subjects, so a model cannot score well by being strong in a single area, and the score is meant to reflect breadth as much as depth. The authors state the requirement plainly: reaching high accuracy on the test demands extensive world knowledge together with problem-solving ability ^[1]. A model that has absorbed a great deal of text but cannot apply it falls short, and so does one that reasons cleanly but knows little about the world.

What they found

The headline result concerned how far the best models were from doing well. Most recent models scored close to random chance — roughly what a system would get by answering without any relevant knowledge at all. The largest GPT-3 model was the exception: it beat random chance by almost 20 percentage points on average ^[1]. That is a real gain, but it still leaves most of the test unanswered in any meaningful sense.

Where the failures were

The more useful findings were about the shape of the failures rather than the average score. On every one of the 57 tasks, the best models still needed substantial improvement before reaching expert-level accuracy ^[1]. Strength in one subject did not carry over into another, so a single overall number hid wide variation between tasks.

The gaps were not confined to obscure subjects. On some socially important topics, such as morality and law, models had near-random accuracy ^[1]. A model that answers legal or ethical questions no better than guessing is a problem for anyone hoping to use it in those areas.

The third finding is about confidence. The authors report that models frequently did not know when they were wrong ^[1]. A model that is equally confident in a correct answer and an incorrect one gives a user no signal about when to check its output, which matters as much as raw accuracy once a system is used for anything consequential.

Why it matters

The paper's contribution is a way of measuring breadth of knowledge and reasoning in one place, rather than one narrow skill at a time. Because the same 57 tasks are given to every model, scores can be compared across systems and across time, and the distance to expert-level performance can be stated rather than assumed.

The results also set a clear baseline. They show that a benchmark spanning many subjects is hard in a way that narrower tests are not: models that looked capable on individual tasks were close to guessing on others, and were poorly placed to report their own uncertainty. Those two observations — uneven knowledge across subjects, and unreliable self-assessment — framed much of what later work on language-model evaluation had to address.