DeepSeek releases DeepSeek-Coder-V2

Background

DeepSeek had released its first code models, DeepSeek Coder, and had since built a general mixture-of-experts model, DeepSeek-V2. Code models are language models trained largely on programming material, and their usefulness depends on how many languages they cover and how much code they can consider at once.

A model that reads a whole project needs a long context window, which is the amount of text it can take into account at one time. A model that works across many programming languages needs broad coverage of syntax and libraries.

What happened

On 17 June 2024 DeepSeek released DeepSeek-Coder-V2, and its paper was posted to arXiv the same day ^[1]. The model was built by continuing pre-training from an intermediate DeepSeek-V2 checkpoint on about 6 trillion additional tokens ^[1]. Continued pre-training means the model kept learning from new data after its earlier training, instead of starting over.

Compared with DeepSeek-Coder, the new model expanded the supported programming languages from 86 to 338 and extended the context window from 16K to 128K tokens ^[1].

Key details

DeepSeek reported that the model performed comparably to GPT-4 Turbo on code tasks, and that it beat GPT-4 Turbo, Claude 3 Opus and Gemini 1.5 Pro on standard coding and math benchmarks ^[1]. These are the company's own reported results on those test sets. The company also reported that the extra training improved coding and math reasoning while keeping general language performance roughly intact ^[1].

On 5 September 2024 DeepSeek merged DeepSeek-Coder-V2-0724 with DeepSeek-V2-0628 into DeepSeek-V2.5 ^[3]. The merge combined the code and chat lines into one model, so the June code release was followed in September by a merged model.

Impact

The release gave developers an open-source code model with a 128K context window and support for 338 programming languages, with a published paper and repository. It also showed that a mixture-of-experts model can be given a large amount of code training while keeping broader language skills. The later merge into DeepSeek-V2.5 shows how the company folded the code model into a single offering.