LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

The problem it targets

Progress in code generation is usually measured with a fixed set of test problems. That works until the problems become famous: once a benchmark is public, its tasks can end up in the training data of later models, and a high score then measures recall rather than ability. The failure is usually called contamination. The authors of LiveCodeBench treat it as the central obstacle to honest evaluation, and they examine a related worry as well — that models may be overfitting to existing benchmarks such as HumanEval and MBPP ^[1]. Their answer is a benchmark that never stops moving. The paper, by Naman Jain, King Han, Alex Gu and seven other authors including Ion Stoica, was submitted to arXiv on 12 March 2024 ^[1].

How it works

LiveCodeBench collects problems continuously, drawing new ones over time from contests on three competitive-programming platforms: LeetCode, AtCoder and CodeForces ^[1]. The design turns the passage of time into a defence. A problem that entered the pool after a model's training data was frozen cannot have been seen by that model, so solving it is evidence of skill rather than memory. At the time of the paper the pool held 400 high-quality problems published between May 2023 and May 2024 ^[1]. Keeping the publication dates also lets the authors sort results by date, so a model can be compared on problems from before and after its training cut-off.

The second design choice is breadth. Writing a function from a natural-language description is only one of the things a coding model is asked to do, so the benchmark adds other scenarios: self-repair, code execution and test-output prediction ^[1]. In self-repair the model is given code that failed and has to fix it; code execution asks what a program does when it runs; test-output prediction asks what a particular test case will produce. A model that writes plausible code but reasons poorly about it will show up differently across these settings.

What the experiments showed

The authors evaluated 18 base models and 34 instruction-tuned models ^[1]. Three threads run through the results: measurements of contamination and what it does to scores, evidence on potential overfitting in existing benchmarks such as HumanEval and MBPP, and comparisons between individual models ^[1]. Comparing a base model with its instruction-tuned counterpart is part of the point, because it separates what pretraining put in from what later tuning added — something a single aggregate score cannot show.

Why it matters

A benchmark that keeps taking in new problems does not go stale the way a frozen one does: the questions stay useful precisely because they keep being replaced. That makes it possible to compare a model released this year with one released later on ground neither of them has seen.

What was released

The authors said they would release all prompts and model completions, along with a general toolkit for adding new scenarios and models ^[2]. Prompts and completions make individual numbers checkable; a toolkit for new scenarios lets other groups extend the evaluation instead of building a parallel one from scratch.