Benchmark contamination

What the term means

A benchmark is only informative if its questions are new to the model being tested. Benchmark contamination describes the case where they are not: the test items, or close variants of them, were present in the model's pretraining data ^[1]. A score then partly reflects recall of text the model has already read rather than the capability the benchmark is supposed to probe, and comparisons between models become hard to interpret.

The difficulty is that contamination leaves no obvious trace. A high score looks the same whether it comes from skill or from memory, and pretraining corpora are typically assembled at a scale where no one has read every document.

Why the problem was investigated

Deng et al. published an investigation in November 2023, motivated by a disparity between inflated benchmark scores and the actual performance of large language models ^[1]. In other words, the prompt for the study was a practical mismatch: models that looked strong on paper did not always behave that way.

The authors note that the problem is especially critical for closed-source models and for open-source models whose training data is not transparent ^[1]. In both cases an outside party cannot simply inspect the pretraining corpus to check whether an evaluation set sits inside it, so contamination has to be inferred rather than observed.

Two ways to detect it

The paper proposes two detection methods, one aimed at open-source models and one applicable to both open and proprietary models ^[1].

The first is a retrieval-based system that explores potential overlaps between evaluation benchmarks and pretraining corpora ^[1]. This approach depends on having the corpus available to search, which is why it suits models whose training data has been released.

The second is a protocol called Testset Slot Guessing, which needs no corpus at all. It masks a wrong answer in a multiple-choice question and prompts the model to fill in the gap, and, separately, it obscures an unlikely word in an evaluation example and asks the model to reproduce it ^[1]. The reasoning behind the design is that a model meeting a genuinely unseen item has little reason to supply precisely the missing option or word, whereas a model that has memorised the item during pretraining can complete it. Because the method only requires querying the model, it can be applied to systems whose weights and data are not public.

What the experiments showed

On the MMLU benchmark, ChatGPT and GPT-4 achieved exact match rates of 52% and 57% respectively when guessing the missing options in benchmark test data ^[1].

On TruthfulQA, the authors found that large language models showed notable performance improvement when provided with additional metadata from the benchmark ^[1].

What the authors argue

The conclusion the authors draw is methodological. They argue that the results call for more robust evaluation methodologies and for benchmarks that are refreshed or held out ^[1]. A refreshed benchmark moves on to new items; a held-out benchmark is kept out of public circulation so that it cannot be absorbed into future pretraining data. Both ideas treat a benchmark less as a fixed exam and more as something that has to be protected and renewed over time.

Limits to keep in mind

Detection of this kind is indirect. Guessing a masked option correctly is evidence that an item, or something like it, was seen before, but such methods produce indications rather than proof, and their strength depends on how the prompts are constructed. The retrieval approach, meanwhile, is only as good as the corpus it is given; where training data is undisclosed, it simply cannot run, which is exactly the gap the guessing protocol was designed to cover. The broader lesson is that a benchmark score is only meaningful together with an account of how the benchmark was kept out of the training data.