DeepSeek releases DeepSeek-V2 with Multi-head Latent Attention

Background

Large language models are expensive to train and to serve. A model's total size affects how much it knows, but its per-token computation and memory use decide how expensive it is to train and serve. Mixture-of-experts designs address this by activating only part of the model for each token, and attention mechanisms address memory, because the memory used to remember earlier tokens grows with every answer.

DeepSeek's earlier work had already explored mixture-of-experts through DeepSeekMoE, and its next large model was built to combine that approach with a new attention design.

What happened

DeepSeek posted the DeepSeek-V2 technical report to arXiv on 7 May 2024 ^[1]. The model is a mixture-of-experts language model with 236B total parameters, of which 21B are activated per token, and a context length of 128K tokens ^[1].

Its most notable design change was Multi-head Latent Attention (MLA). MLA compresses the key-value cache into a latent vector, which reduces the memory needed while the model generates text. The model also used DeepSeekMoE for economical sparse training ^[1].

Key details

The model was pre-trained on 8.1 trillion tokens and then went through supervised fine-tuning and reinforcement learning ^[1]. Supervised fine-tuning teaches the model from example answers, and reinforcement learning adjusts it based on which outputs are rewarded.

Compared with DeepSeek 67B, DeepSeek reported a 42.5% cut in training costs, a 93.3% reduction in the key-value cache and a maximum generation throughput of 5.76 times ^[1]. The company said the model reached top-tier performance among open-source models with only 21B activated parameters ^[1]. These are the company's own figures, and they describe the design's efficiency rather than a single benchmark score.

Impact

DeepSeek-V2 showed that a model with a very large total parameter count could be run with a much smaller per-token computation and memory budget. The report's two main architectural ideas, MLA and the use of DeepSeekMoE, are what its efficiency claims rest on. Readers who follow the history of efficient large models can treat this report as a clear example of attention and mixture-of-experts being combined in one open design.