MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

MMLU-Pro is a benchmark for multi-task language understanding. It comes from a paper submitted to arXiv on 3 June 2024 by Yubo Wang, Xueguang Ma, Ge Zhang and 14 other authors, among them Wenhu Chen ^[1]. Its purpose is stated in the title: to be a more robust and more challenging successor to the MMLU benchmark.

The problem it set out to solve

MMLU is a mostly knowledge-driven test ^[1]. A model that has absorbed enough facts can pick the right option out of four without doing much reasoning, so the score says more about stored knowledge than about thinking. The original set also contains questions that are trivial or noisy — too easy, or flawed in ways that make the resulting score hard to interpret ^[1]. Once models score near the top of such a benchmark, the number stops telling us anything useful.

How the benchmark was built

MMLU-Pro keeps the multi-task, multiple-choice format but changes what the questions demand: it integrates more challenging, reasoning-focused questions into the set ^[1]. The choice set is expanded from four options to ten ^[1]. That single change raises the cost of guessing, because the ten options carry three times as many distractors as MMLU, so wrong answers are more plausible and elimination is harder ^[2].

Construction also involved cleaning. The authors filtered out 5,886 questions across the various subjects ^[2], as part of eliminating trivial and noisy items from the original benchmark ^[1]. The final dataset comprises 14 discipline subsets with 12,032 questions in total ^[2].

What the results showed

The paper reports two kinds of measurement. The first is difficulty: compared with MMLU, accuracy on MMLU-Pro drops by 16% to 33% ^[1]. A model that looks close to solved on the older benchmark still has a large gap on the new one.

The second is stability under prompting. The authors tested 24 different prompt styles and looked at how much a model's score moved when only the prompt changed. In MMLU that sensitivity was 4–5%; in MMLU-Pro it fell to about 2% ^[1]. Scores therefore depend less on how a question happens to be phrased.

Why it matters

A benchmark is useful when it is hard enough that strong models still fail some items, when guessing does not pay off, and when the score does not swing with prompt formatting. MMLU-Pro is designed against all three problems: reasoning-heavy questions, ten options, and a measured reduction in prompt sensitivity ^[1][2]. For anyone comparing models, the design intent is that a score on this benchmark reflects reasoning ability more than recall of facts, and is more reproducible across evaluation setups.