DeepSeek Coder

What it is

DeepSeek Coder is a family of open-source models for programming, and it was DeepSeek's first model release, dated 2 November 2023^[3]. The family covers a wide range of sizes: the technical report describes models from 1.3B to 33B parameters^[1], while the GitHub repository lists 1B, 5.7B, 6.7B and 33B versions^[2]. Each of them was trained from scratch on 2 trillion tokens rather than being adapted from a general-purpose model^[1]. A code model is a language model trained mostly or entirely on source code so that it can complete and write programs; "from scratch" means the parameters began from random initialisation and the whole model was fitted to code data, instead of starting from a model that had already learned ordinary text.

How it was trained

Pre-training used a project-level code corpus: the training data was organised around whole software projects rather than isolated files, so the model saw how code is laid out and referenced across a repository^[1]. Two design choices matter for how the models behave afterwards. First, training included an extra fill-in-the-blank task, in which a stretch of code is hidden and the model has to reconstruct it from the surrounding text; this is what makes code infilling work, meaning completion in the middle of a file rather than only at its end^[1]. Second, the models were given a 16K-token context window, so a prompt can carry a fairly large amount of surrounding code at once^[1].

Results, licensing and variants

DeepSeek reported state-of-the-art results among open-source code models, and said the models outperformed the closed-source Codex and GPT-3.5 on multiple benchmarks^[1]. They were released under a permissive license that allows research and unrestricted commercial use^[1]. The GitHub repository is licensed under MIT, with the model weights governed by a separate Model License that supports commercial use^[2]. Alongside the base models, instruction-tuned versions such as deepseek-coder-6.7b-instruct and deepseek-coder-33b-instruct were produced by instruction fine-tuning, the extra training step in which a model learns to follow requests rather than only continue text^[2]. The technical report was posted to arXiv on 25 January 2024^[1].

Where it sits in the line

DeepSeek-Coder-Base-v1.5 7B was later the starting checkpoint for DeepSeekMath 7B, so the code models fed directly into DeepSeek's later work on mathematical reasoning^[4]. The successor, DeepSeek-Coder-V2 in June 2024, expanded the number of supported programming languages from 86 to 338 and the context window from 16K to 128K tokens^[5]. Wikipedia lists the original DeepSeek-Coder line as discontinued^[3].