DeepMind introduces AlphaCode for competitive programming

Competitive programming is a form of contest in which participants receive a problem statement and a time limit, and must write a program that produces correct output on hidden test inputs. Solutions are judged automatically, and the ranking depends both on how many problems a contestant solves and on how quickly they solve them. Because the problems are unseen and the programs have to be exactly right, such contests are a demanding test of the reasoning that goes into writing code.

What DeepMind announced

On 2 February 2022, DeepMind introduced AlphaCode, a system that writes computer programs at a competitive level.^[1] DeepMind said AlphaCode was the first AI code generation system to reach a competitive level of performance in programming competitions.^[1]

How it was trained

AlphaCode was pre-trained on selected public GitHub code and then fine-tuned on a relatively small competitive programming dataset.^[1] The two stages do different jobs. Pre-training exposes the system to a large body of existing code, so that it learns the patterns and conventions of programs written by people. Fine-tuning then adapts it to the narrower setting of contest problems, using a dataset that is small compared with the material used in the first stage.

How it produced answers

Rather than committing to a single program, AlphaCode generated many candidate solutions and filtered them before submitting up to 10.^[1] The filtering step matters because a contest accepts only a small number of submissions. Generating a large pool of candidates raises the chance that at least one of them is correct, but the judge sees only a handful, so the system has to narrow the pool down first.

Results

DeepMind evaluated AlphaCode by simulating participation in 10 recent Codeforces contests, each newer than its training data, with Codeforces' permission.^[1] On average across those 10 contests, it ranked within the top 54% of participants.^[1] Because every contest used in the evaluation was held after the system's training data had been fixed, the comparison was made against problems the system had not seen.

Publication and released data

The peer-reviewed paper appeared on the cover of Science on 8 December 2022.^[1] DeepMind also released the CodeContests dataset on GitHub.^[1] Publishing the dataset gives other researchers the same competitive programming material to work with, so that results on this kind of task can be produced and checked outside the original team.

Why it matters

Writing a program that passes hidden tests is different from writing text that looks plausible. The output either compiles and produces correct answers on every test, or it does not. AlphaCode's results were reported in terms of placement in real contests against human participants, and its performance was measured on contests held after its training data, which is why DeepMind described the system as reaching a competitive level rather than simply producing code that looks reasonable.^[1] The release of the CodeContests dataset put the evaluation material in public hands alongside the paper.^[1]