AlphaCode

AlphaCode is a code-generation system built by Google DeepMind for competitive programming. Given a problem statement of the kind used in online contests, it writes a program — in C++ or Python — meant to be submitted and judged against hidden tests. DeepMind first announced the system on 2 February 2022.^[1]

How it works

The system does not try to write one careful solution. For each problem it generates a very large number of candidate programs in C++ and Python, orders of magnitude more than earlier work on code generation.^[1] That pool is then reduced sharply: programs are filtered on their behaviour, clustered and reranked, leaving a small set of 10 candidates to submit.^[1] Filtering on behaviour means looking at what a program actually does when it runs — for example whether it produces the expected output on the sample tests shipped with a problem — rather than at how the code looks; clustering then keeps the final submissions from being ten variations on a single idea. The paper credits three ingredients: a large and clean competitive-programming dataset, efficient transformer architectures, and large-scale sampling combined with behaviour-based filtering.^[3] Model sampling is followed by filtering on program behaviour, down to a small set of submissions.^[3]

Results

AlphaCode was evaluated on 10 recent Codeforces contests that were held after its training data was collected, so the problems were new to it. On those contests it achieved an estimated rank within the top 54% of participants, about the level of the median competitor.^[1] In simulated Codeforces evaluations it averaged a ranking in the top 54.3% in competitions with more than 5,000 participants.^[3]

The dataset

Alongside the system, DeepMind released CodeContests on GitHub: a competitive-programming dataset of problems, solutions and tests.^[1] The company said the dataset includes extensive tests, which it described as a critical feature that current datasets lack.^[1]

Publication

The arXiv preprint "Competition-Level Code Generation with AlphaCode" was submitted on 8 February 2022.^[3] The paper has 26 authors, including Oriol Vinyals.^[3] It was published on the cover of Science on 8 December 2022.^[1]

Later work

In December 2023, Google announced AlphaCode 2 together with Gemini. AlphaCode 2 is built on a specialised version of Gemini, and was estimated to perform better than 85% of competition participants.^[2]

Why it matters

Competitive programming is a demanding setting for code generation. A solution has to be correct on tests the solver cannot see, and it has to run fast enough and use little enough memory to pass within contest limits, so a program that merely looks plausible is not enough. Putting a model into that setting and measuring it against human contestants produced a concrete comparison point: an estimated rank near the middle of the field on contests the system had never seen.^[1] The work also left behind CodeContests, an openly released dataset with extensive tests, which gives later work something to train and evaluate on.^[1]