OpenAI opens Codex to developers in its API

Background

Until the summer of 2021, OpenAI's API had been built around text models. The company launched its GPT-3 text models in June 2020, and developers used them for language tasks rather than for producing working programs^[1]. A language model of this kind is trained to continue a piece of text: given what came before, it estimates what should come next, token by token. Getting such a model to write code means steering that same ability toward a different kind of text.

A GPT model trained on code

OpenAI's research paper on Codex was submitted on 7 July 2021 and described a GPT model fine-tuned on public code from GitHub^[2]. Fine-tuning means starting from a model that has already learned from a large body of text and continuing to train it on a narrower, more specific dataset. Here the narrower dataset was source code.

OpenAI described Codex as a descendant of GPT-3 whose training data contained both natural language and billions of lines of public source code^[4]. Mixing the two matters for the intended use: a user states what they want in ordinary words, and the model has to connect that description to the syntax and idioms of a programming language.

The release

On 10 August 2021 OpenAI released an improved version of Codex through its API in private beta^[1]. The release extended the company's API beyond the text models it had offered since June 2020^[1]. The label private beta indicated that access was limited rather than general.

According to OpenAI, Codex was proficient in more than a dozen programming languages, and could interpret simple commands written in natural language and execute them on the user's behalf^[4]. Both halves of that description belong to the same claim: understanding the request, and turning it into code that does the job.

HumanEval and the numbers

Alongside the model, the paper introduced a benchmark called HumanEval, and reported Codex's results on it: the model solved 28.8% of the problems when allowed a single attempt, and 70.2% when 100 samples were drawn^[2]. The gap between the two figures reflects how the second setting works — a model can be sampled many times, and a problem counts as solved if any of the samples is correct. Reporting both numbers makes visible how much of the performance depends on generating candidates and checking them, rather than on getting the answer right on the first try.

Connection to GitHub Copilot

OpenAI said that Codex is the model that powers GitHub Copilot, which OpenAI built and launched with GitHub a month earlier^[4]. The same underlying capability therefore appeared in two places at once: a general API that developers could call from their own programs, and a product made jointly with GitHub.

Later

Codex models stayed available in OpenAI's API for a time and were shut down on 23 March 2023^[3]. By then the idea the release had put in front of developers — describing a task in ordinary language and having a model write the code — had become familiar enough that the original models could be retired.