Holistic Evaluation of Language Models (HELM)

The problem: scores that cannot be compared

Language models are usually judged by benchmarks, but before this work the benchmarks rarely lined up. HELM's authors measured the overlap and found that models had on average been evaluated on just 17.9% of the scenarios in HELM's core set, and that some prominent models did not share a single scenario in common ^[1]. When two models share no task, their reported numbers cannot be set side by side, however precise each number looks. The paper therefore treats evaluation as a design problem: decide in advance which scenarios and which metrics matter, then run every model on the same ones under standardized conditions.

A taxonomy of scenarios and metrics

HELM starts by taxonomizing the space of possible scenarios and metrics for language models, so that the choice of what to measure is explicit rather than inherited from whichever benchmarks happen to be popular ^[1]. From that space it adopts a multi-metric approach: for each of 16 core scenarios it measures 7 metrics whenever possible — accuracy, calibration, robustness, fairness, bias, toxicity and efficiency — which was possible 87.5% of the time ^[1].

The seven metrics ask different questions about the same model. Accuracy asks whether the answer is correct. Calibration asks whether the confidence the model expresses matches how often it is actually right, so that a model saying it is 90% sure should be right about nine times in ten. Robustness asks whether small, meaning-preserving changes to the input change the output. Fairness and bias ask whether the model behaves differently for different groups of people, or reproduces stereotypes. Toxicity asks whether it produces harmful text. Efficiency asks what it costs to run. A single accuracy number hides all of this.

What was actually run

The authors evaluated 30 prominent language models, spanning open, limited-access and closed models, on 42 scenarios ^[1]. Twenty-one of those 42 scenarios had not previously been used in mainstream language-model evaluation, so the study deliberately reached beyond the standard benchmark set ^[1]. On top of the main run, HELM added 7 targeted evaluations built on 26 targeted scenarios, in order to analyse specific aspects in more depth ^[1][2].

Coverage and findings

Comparability came from standardization: coverage of the core scenarios rose from 17.9% to 96.0%, so that all 30 models were benchmarked on the same core scenarios and metrics under standardized conditions ^[1]. That is what makes a table of results meaningful — a blank cell now means a model was not run, not that a different task was substituted for it. The evaluation surfaced 25 top-level findings ^[1].

Open artifacts

Alongside the paper, the authors released all raw model prompts and completions publicly, together with a general modular toolkit ^[1]. Releasing prompts and outputs matters because scores alone do not show what the model was asked. With the raw text, others can inspect failures, re-score the same outputs with a different metric, or check whether a particular prompt was doing the work.

Why it matters, and what it does not settle

HELM's contribution is less a verdict on which model is best than a change in the shape of the question: not "which model is best" but "which model is better at what, and at what cost." When accuracy, robustness, fairness and efficiency are reported together, the result is often a set of trade-offs rather than a single ranking; a model that leads on accuracy may pay for it in efficiency or robustness. Such an evaluation also has boundaries. It is a snapshot of a set of models at one moment, and models change. Fairness and toxicity are hard to compress into a single number, and the choice of scenarios is itself a judgement: what is included becomes visible, and what is left out does not.