DeepSeek-Coder-V2

What it is

DeepSeek-Coder-V2 is an open-source code model that DeepSeek built on a mixture-of-experts (MoE) design. In an MoE model, each input token is routed to only a small part of the network, so the total number of parameters can be far larger than the number actually used for any single token. The project's repository describes two sizes: a Lite version with 16B total parameters, of which 2.4B are activated per token, and a full version with 236B total and 21B activated. Both are listed with a 128K-token context window.^[2] Like other code models, it is meant to work on source code rather than ordinary prose: completing a function, filling in a gap, or answering questions about a file.

Where it came from

The model was released on 17 June 2024, the same day its paper was posted to arXiv.^[1] Rather than training a code model from scratch, DeepSeek continued pre-training from an intermediate checkpoint of its general-purpose DeepSeek-V2 model, using about 6 trillion additional tokens.^[1] That starting point explains the shape of the release: the model inherits a general language model's structure and is then pushed toward code and mathematics rather than being built only for programming from the beginning.

What changed from DeepSeek-Coder

Compared with the earlier DeepSeek-Coder, the number of supported programming languages went from 86 to 338, and the context window grew from 16K to 128K tokens.^[1] A longer context matters for code because a repository, a long file, or a change spread across several files may simply not fit into a short window, so the model can see less of the problem it is asked to work on.

Results

DeepSeek reported performance comparable to GPT-4 Turbo on code tasks, and better results than GPT-4 Turbo, Claude 3 Opus and Gemini 1.5 Pro on standard coding and math benchmarks.^[1] The extra pre-training improved coding and mathematical reasoning while general language performance stayed roughly intact.^[1] That last point is the usual worry when a model is trained further on a narrow domain: it can get better at the target task while losing some of its handling of ordinary text, and here the loss was reported as small.

The repository's README lists 90.2 on HumanEval, a benchmark built from short Python programming problems, and 75.7 on MATH, a set of competition-style mathematics problems, for DeepSeek-Coder-V2-Instruct.^[2]

Licensing

The code repository is released under the MIT licence, and the model series supports commercial use under a separate Model License.^[2]

Where it sits

The model's base version was also used to upgrade DeepSeek-V2-Chat in June 2024.^[3] An updated DeepSeek-Coder-V2-0724 followed in July 2024,^[3] and on 5 September 2024 DeepSeek merged DeepSeek-Coder-V2-0724 with DeepSeek-V2-0628 into a single model, DeepSeek-V2.5.^[3] Wikipedia groups Coder-V2 with the DeepSeek-V2 line, which has since been discontinued.^[4] The sequence shows a recurring pattern: a code-specialised model is trained on top of a general one, then folded back into the next general release instead of living on as a separate line.