DeepSeek-V2

What it is

DeepSeek-V2 is a large language model built on a mixture-of-experts (MoE) design. It holds 236B total parameters, but only 21B of them are activated for each token it generates, and it accepts contexts of up to 128K tokens^[1]. The MoE idea is that a model can carry far more parameters than it uses at any one moment: a routing step sends each token to a small subset of expert sub-networks, so total capacity grows without a proportional rise in the compute spent per token. DeepSeek's claim for the release was that this budget was enough to reach top-tier performance among open-source models^[1].

Where it came from

DeepSeek announced the release on 6 May 2024, and the technical report went up on arXiv on 7 May 2024^[1][2]. A smaller sibling, DeepSeek-V2-Lite, followed on 16 May 2024. Lite keeps the family design at a much smaller scale: 16B total parameters with 2.4B activated per token, and a 32K-token context instead of 128K^[2].

What was new in the architecture

Two design choices carried most of the weight. The first is Multi-head Latent Attention (MLA), which compresses the key-value cache into a latent vector. That cache is the store of keys and values a model keeps so it can attend to earlier tokens without recomputing them; the longer the context, the more memory it occupies, so its size largely decides what a serving system can afford to keep in memory. The second is DeepSeekMoE, the sparse layout the model uses to keep training economical^[1].

Training ran in stages: pre-training on 8.1 trillion tokens, then supervised fine-tuning and reinforcement learning^[1].

The efficiency figures were measured against DeepSeek 67B. DeepSeek reported that V2 cut training costs by 42.5%, shrank the key-value cache by 93.3%, and raised maximum generation throughput to 5.76 times the earlier figure^[1]. These are the model's own comparisons rather than independent benchmarks.

How the line developed

V2 did not stand still after launch. DeepSeek-V2-Chat was upgraded onto the Coder-V2 base in June 2024, producing DeepSeek-V2-Chat-0628^[3]. On 5 September 2024 came DeepSeek-V2.5, which merged DeepSeek-V2-0628 with DeepSeek-Coder-V2-0724^[3]. By the end of the run, the V2 series had received five updates between May and December 2024, the last of them DeepSeek-V2.5-1210^[4].

Why it matters

V2 is where the architecture DeepSeek went on to use was first validated. The DeepSeek-V3 technical report states that its MLA and DeepSeekMoE design was first validated in DeepSeek-V2^[5]. That makes V2 the point at which the combination of sparse experts and latent attention turned from a design into a working model that others could measure against.