OpenAI Codex (2021 model)

What it is

OpenAI Codex is a GPT language model fine-tuned on publicly available code from GitHub. It was introduced in the paper "Evaluating Large Language Models Trained on Code", submitted to arXiv on 7 July 2021 ^[1]. The paper lists 58 authors, led by Mark Chen, Jerry Tworek and Heewoo Jun, and ends with Ilya Sutskever and Wojciech Zaremba ^[1].

Codex is a system that translates natural language into code. A developer can describe what a program should do, and the model writes the code. Fine-tuning means taking a model that has already been trained on general text and training it further on a specific kind of data, here code.

Origins

The paper introduced HumanEval, a new benchmark that measures the functional correctness of programs synthesised from docstrings. A docstring is a short written description of what a function should do, placed inside the code. Checking whether the program actually works is a stricter test than checking whether it looks plausible ^[1].

On HumanEval, Codex solved 28.8% of problems, against 0% for GPT-3 and 11.4% for GPT-J ^[1]. The gap with GPT-3 shows how much the code training mattered for this task.

Repeated sampling proved surprisingly effective. With 100 samples per problem, Codex produced a working solution for 70.2% of problems ^[1]. This means that the model can generate several attempts and that a correct one is often among them.

Main features

On 10 August 2021, OpenAI released an improved version of Codex through its API in private beta ^[2]. The paper had been submitted a month earlier, on 7 July 2021.

The paper noted limitations, such as difficulty with long chains of operations and with binding operations to variables. It also discussed safety, security and economic impacts ^[1].

The API models included code-davinci-001, code-davinci-002, code-cushman-001 and code-cushman-002. code-davinci-002 was released on 15 March 2022 with edit and insert capabilities, and it was part of the GPT-3.5 series ^[3][4].

Retirement and later use

OpenAI announced the deprecation of the Codex API models on 20 March 2023, and shut them down on 23 March 2023 ^[3].

The name Codex was later reused for OpenAI's coding agent, launched in 2025 ^[5]. The two are different products: the original Codex was a model offered through an API, while the later product is a coding agent.

Why it matters

Codex showed that a language model trained on public code could write working programs for a standard test, and it made that ability available to developers through an API. Readers who follow later coding tools can trace part of their history back to this model and paper.