Open LLM Leaderboard

What it was

The Open LLM Leaderboard was a public leaderboard run by Hugging Face that ranked large language models by their scores on a fixed set of benchmarks. Its premise was comparability: rather than collecting numbers that model authors reported in their own papers, the team ran the evaluations itself, in one identical setup, so that results were reproducible and could be lined up directly against each other ^[1].

Why it started

The leaderboard was created in 2023 by Hugging Face's RLHF team. The motivation came out of a practical problem: the team found it nearly impossible to reproduce and compare the scores published with new models. Papers described different evaluation pipelines, so a claimed result was hard to check and hard to set beside a competing model. Their answer was to evaluate reference models in the exact same setup, which made the results fully reproducible and comparable ^[1].

The format caught on. In the ten months before the second version was announced, the leaderboard was visited by more than 2 million unique people, and around 300,000 community members used and collaborated on it each month ^[1].

Saturation and version 2

By mid-2024 the original benchmarks had been overused and had become saturated. Models were reaching baseline human performance on HellaSwag, MMLU and ARC, which meant the tests could no longer separate strong models from one another. Some scores showed signs of contamination, and some benchmarks contained errors — in GSM8K, for example, a specific end-of-generation token unfairly penalized verbose models ^[1].

Hugging Face announced version 2 on June 26, 2024. The announcement was by Clementine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer and Thomas Wolf, and it replaced the old tests with six new benchmarks: MMLU-Pro, GPQA, MuSR, MATH Level 5, IFEval and BBH. Scores were normalized between the random baseline and the maximum score, so a number on the leaderboard reflected how far a model had moved from guessing toward the ceiling ^[1].

The evaluations ran on Hugging Face infrastructure with 300 Nvidia H100 GPUs, using an updated evaluation suite developed in collaboration with EleutherAI ^[2].

Retirement

The leaderboard was retired two years after it started. On March 14, 2025, maintainer Clementine Fourrier announced that it was officially retiring, after evaluations of more than 13,000 models. The reasoning was that benchmarks have to follow along as model capabilities change: the leaderboard was slowly becoming obsolete, and it could encourage people to hill climb directions that were no longer relevant ^[3].

By then the surrounding landscape had changed. More than 200 community-led leaderboards were available on Hugging Face, and the team's ongoing evaluation work continued under the Open Evals hub organization ^[3].

Where it sits

What the Open LLM Leaderboard left behind was less a single score than a piece of shared infrastructure: during the years when open models multiplied, it gave them one common reference point to be measured against. Its own arc — created, saturated, replaced, retired — also illustrates the point its maintainers made at the end. Models improve, benchmarks age, and evaluation has to keep moving with them ^[3].