DeepSeek releases its first model, DeepSeek Coder

Background

DeepSeek is a Chinese AI company, and its first public model was a code model. Code models are language models trained on source code. They can write functions from a natural-language description, complete a line the programmer has started, or explain what a piece of code does. Such tools became a common use of large language models because programmers can check the output by running it.

An open-source release matters to developers because the trained weights are downloadable. Researchers can study the model, companies can run it on their own servers, and small teams can adapt it without paying a service provider for every request.

What happened

On 2 November 2023 DeepSeek released DeepSeek Coder ^[3]. The release was a family of models rather than a single model. The sizes ranged from 1.3B to 33B parameters, and each one was trained from scratch on 2 trillion tokens ^[1]. Training from scratch means the training began from randomly set initial values, rather than from the weights of another model.

The models were built for programming. Their training combined a project-level code corpus with a fill-in-the-blank task, which lets the model fill a gap between code written before and after it. The models were trained with a 16K context window ^[1].

Key details

DeepSeek reported that the models were state-of-the-art among open-source code models, and that they outperformed the closed-source Codex and GPT-3.5 on several benchmarks ^[1]. These are the company's own reported results, measured on standard test sets. Readers can compare them with later independent evaluations.

The release was made under a permissive license that allows research and unrestricted commercial use ^[1]. The technical report was posted to arXiv on 25 January 2024, about three months after the models were released ^[1].

Impact

The release gave developers an openly available code model from a new Chinese lab, with a clear license and a published technical description. It also established the family that DeepSeek later extended with more capable code models. The original models were later described as discontinued ^[3]. The event is best read as the starting point of DeepSeek's public model history rather than as a single product launch.