DeepSeek-V3 Technical Report

What the report sets out to do

A large language model is usually judged on two things that pull against each other: how well it performs, and how much computation it takes to build. The DeepSeek-V3 Technical Report, credited to DeepSeek-AI and submitted to arXiv on 27 December 2024 as 2412.19437, is organised largely around the second of those. It documents the architecture, the training process and the evaluations of DeepSeek-V3, a mixture-of-experts language model with 671 billion total parameters, of which 37 billion are activated for each token.^[1]

Mixture of experts is a way of making a very large model cheaper to run. The network contains many separately parameterised sub-networks, called experts, and for each token only a small subset of them is used. The total parameter count therefore describes how big the model is, while the activated count describes how much computation each token actually costs. That gap is what allows a model of this size to be trained and served at all.

Architecture

DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and the DeepSeekMoE architecture. Neither is presented as new here: the report says both were thoroughly validated in DeepSeek-V2, the team's earlier model, so the new work builds on components that had already been tested.^[1]

Two design choices in the report are described as firsts for the model. The first is an auxiliary-loss-free strategy for load balancing. In a mixture-of-experts model, training has to keep tokens spread reasonably across the experts, or a few experts get all the traffic while the rest go unused. The usual way of doing this is to add an auxiliary loss term that penalises imbalance, which means the training objective is no longer purely the language-modelling objective. The strategy described here balances expert load without such a term. The second is a multi-token prediction training objective, in which the model is trained to predict more than one upcoming token at a time rather than only the single next one.^[1]

Training

Pre-training used 14.8 trillion tokens. It was followed by supervised fine-tuning and reinforcement learning stages.^[1]

The report gives unusually detailed efficiency figures. The full training run required only 2.788 million H800 GPU hours.^[1] The pre-training stage alone accounted for 2.664 million of those hours and finished in under two months on a cluster of 2,048 H800 GPUs.^[1] The authors also report that training was stable throughout, with no irrecoverable loss spikes and no rollbacks, meaning the run did not have to be restarted from an earlier checkpoint after a divergence in the loss.^[1]

A further claim concerns numerical precision. The report says it validates FP8 mixed-precision training at extremely large scale for the first time: that is, carrying out the bulk of the computation in a lower-precision 8-bit floating-point format while keeping the run numerically stable, rather than relying on the higher-precision formats more commonly used for training at this size. That validation rests on the pre-training run described above.^[1]

Results and release

The report's evaluations find that DeepSeek-V3 outperforms other open-source models and reaches performance comparable to leading closed-source models.^[1] Model checkpoints are published in the deepseek-ai/DeepSeek-V3 repository on GitHub, so the trained weights are available outside the team that built them.^[1]

Why it matters

The report is useful less as a claim about a single model than as a documented account of how far training efficiency can be pushed. Three things in it are relevant to anyone building or studying models at this scale. One is the ratio between capability and compute: 2.788 million H800 GPU hours for a model with 671 billion total parameters, with the whole pre-training stage completed in under two months on 2,048 GPUs.^[1] Another is the demonstration that an 8-bit floating-point format can carry a pre-training run of this size without loss instability.^[1] The third is the architecture-level work on load balancing and multi-token prediction, which addresses problems that appear in any mixture-of-experts training run rather than only in this one.^[1]

Because the checkpoints are public, the report can be checked against the model itself rather than read as a description of something unavailable.^[1] That combination — a detailed account of the training process, efficiency numbers stated in GPU hours, and released weights — is what makes the document a reference point for later work on training large models economically.