DeepSeekMath

What it is

DeepSeekMath is a 7-billion-parameter open language model trained for mathematical reasoning. The paper describing it was posted to arXiv on 5 February 2024 ^[1]; Wikipedia records the DeepSeek-Math release in April 2024 ^[2].

How it was trained

DeepSeekMath was not trained from scratch. The team took DeepSeek-Coder-Base-v1.5 7B, an existing code model, and continued pre-training it on 120 billion math-related tokens mined from Common Crawl, together with natural-language and code data ^[1]. Those 120 billion tokens were collected from 35.5 million mathematical web pages extracted from Common Crawl, using a fastText-based classifier run over four iterations ^[1].

Before training, the data was decontaminated: any text segment containing a 10-gram that exactly matches an evaluation benchmark was removed ^[1]. This matters because math benchmarks circulate widely online, and a model that has seen test items during training can score well without actually reasoning better.

The authors attributed the gains to two things: the data-selection pipeline for public web data, and a new reinforcement-learning algorithm ^[1].

Results

The paper reports scores for two variants. DeepSeekMath-Base 7B scored 64.2% on GSM8K and 36.2% on MATH ^[1]. DeepSeekMath-Instruct 7B, the version tuned for following instructions, scored 82.9% on GSM8K and 46.8% on MATH ^[1].

The headline number is on MATH, which the paper describes as a competition-level benchmark: 51.7% without external toolkits or voting, and 60.9% when using self-consistency over 64 samples ^[1]. Self-consistency means sampling many answers and taking the majority result, so it costs far more computation than a single pass. GSM8K, a set of school-level word problems, is the easier of the two benchmarks; MATH is harder, which is why MATH scores are watched more closely as a sign of real mathematical reasoning.

GRPO

The algorithmic contribution is Group Relative Policy Optimization (GRPO), a variant of PPO ^[1]. PPO is a widely used policy-gradient method in reinforcement learning; in practice it usually keeps a separate value function that estimates how much better a given action is than average, and that extra model is part of why PPO is memory-hungry. The "group relative" in the name refers to comparing sampled candidates within a group to establish the baseline for an update. According to the paper, GRPO improves mathematical reasoning while using less memory than PPO ^[1]. Memory is one of the practical limits on this kind of training, so a cheaper update rule changes what can be done with a given amount of hardware.

Where it sits

DeepSeekMath is one of DeepSeek's early specialized models: it starts from a code model, continues pre-training toward mathematics, and then adds reinforcement learning. The line continued afterwards, with DeepSeek-Prover-V2 in April 2025 and DeepSeekMath-V2 in November 2025 ^[2].