DeepSeek releases DeepSeek-V3

Background

DeepSeek had already validated two architecture techniques, Multi-head Latent Attention (MLA) and DeepSeekMoE, in its earlier DeepSeek-V2 model. DeepSeek-V3 kept that architecture and added new training methods on top of it. ^[2]

What happened

On 26 December 2024 DeepSeek released DeepSeek-V3 and said it made the models and the paper fully open source. The technical report was posted to arXiv on 27 December 2024. ^[1][2]

Key details

The model has 671 billion total parameters, of which 37 billion are activated for each token, and supports a 128K-token context. ^[1]

It was pre-trained on 14.8 trillion tokens and then refined with supervised fine-tuning and reinforcement learning. ^[2]

DeepSeek reported that full training took 2.788 million H800 GPU hours, using an FP8 mixed-precision training framework, with no irrecoverable loss spikes and no rollbacks. ^[2]

DeepSeek said the model generated 60 tokens per second, three times faster than V2. ^[1]

The code is MIT-licensed, and the model license supports commercial use. API prices from 8 February 2025 were $0.27 per million input tokens on a cache miss, $0.07 on a cache hit and $1.10 per million output tokens. ^[1][3]

In the chat-model results DeepSeek reported, DeepSeek-V3 scored 90.2 on MATH-500, against 74.6 for GPT-4o-0513 and 78.3 for Claude-3.5-Sonnet-1022. On SWE-bench Verified it scored 42.0, behind Claude-3.5-Sonnet-1022 at 50.8. ^[2]

A mixture-of-experts model can be pictured as one network made of many specialist parts, with a routing step that decides which parts handle each token. The model holds far more parameters than it uses on any single token, so the active-parameter count is the better guide to how much work each token requires.

Impact

The release put a very large open-weight model, its full technical report and the cost figures DeepSeek reported into public view at the same time. The comparisons above are DeepSeek's own reported results. ^[2]