Evaluating Large Language Models Trained on Code (Codex / HumanEval)

The problem it addresses

A language model trained on text learns to predict what comes next, and when that text is source code, the predictions often look like real programs. Looking right is not the same as working. The usual way of scoring generated text, comparing it with a reference answer, is a poor fit for code: two programs can be written quite differently and both be correct, while a program that closely resembles a reference can still fail on the inputs that matter. The paper, by Mark Chen, Jerry Tworek, Heewoo Jun and a team of more than 50 OpenAI authors, was submitted to arXiv on July 7, 2021, and addresses both halves of this problem at once: a model trained on code, and a way of measuring whether what it writes actually runs correctly ^[1].

Codex and HumanEval

Codex is a GPT language model fine-tuned on publicly available code from GitHub, and the paper studies its ability to write Python ^[1]. Fine-tuning means starting from a model that has already learned general language statistics and continuing to train it on a narrower body of text, so that its predictions shift toward the patterns in that body. In this case, the model sees large amounts of real source code rather than mostly prose.

To evaluate the results, the authors released HumanEval, a new evaluation set that measures functional correctness for synthesizing programs from docstrings ^[1]. The setup is a familiar one for programmers: the model is given a docstring, the comment block that describes what a function is supposed to do, and must write the function itself. Because the criterion is functional correctness, the generated program is assessed on whether it behaves correctly rather than on how it is written. That makes the benchmark a test of usable output rather than of surface similarity.

Results

On HumanEval, Codex solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4% ^[1]. The comparison is informative because the three models differ mainly in how much code they were trained on: the general-purpose GPT-3 essentially never produces a working solution, and GPT-J, which is not the specialized model here, gets roughly one problem in nine.

A second result concerns how the model is used. Repeated sampling from the model turns out to be a surprisingly effective strategy for producing working solutions: with 100 samples per problem, Codex solves 70.2% of them ^[1]. In other words, asking many times and keeping a candidate that works does far better than relying on a single attempt. The finding depends on having some way to tell a working solution from a broken one, which is exactly what the functional-correctness framing supplies: a candidate can be checked by running it.

Limitations and broader impacts

The paper is explicit about where the model struggles. It has difficulty with docstrings that describe long chains of operations, and with binding operations to variables ^[1]. Both are cases where the specification is spread across several steps and the model must keep track of intermediate results and connect them to names, rather than producing one self-contained expression. The authors also discuss the broader impacts of code generation, covering safety, security, and economics ^[1].

Why it matters

Two things from this work outlast the specific numbers. The first is HumanEval itself: by tying evaluation to functional correctness, it gives code generation a score that means something concrete, so that models can be compared on whether their programs work ^[1]. The second is reach. A distinct production version of Codex powers GitHub Copilot ^[1], which puts model-written code directly into the editing environment where programmers work. The paper's sampling result points at how such a system is likely to be used in practice: not as an oracle that answers once and is trusted, but as a generator of candidates that a person, or a test suite, then checks.