DeepSeekMath introduces the GRPO reinforcement-learning algorithm

Background

Competition mathematics is a demanding test for language models. The problems are built so that the answer cannot simply be recalled; they need several steps of reasoning, and one slip in the middle usually ruins the final result. The MATH benchmark used in this work is of exactly that kind, made of competition-level problems ^[1]. One common way to improve a model on such tasks is reinforcement learning: the model produces answers, each answer is scored, and training pushes the model toward the outputs that scored well. The standard algorithm for this in language-model training has been Proximal Policy Optimization (PPO). PPO relies on a learned value function to estimate how good a given situation is, and in practice that value function is typically another model of comparable size to the policy model itself ^[1]. Carrying a second model of that size through training is expensive in memory.

What was released

A paper posted to arXiv on 5 February 2024 introduced DeepSeekMath 7B, a 7-billion-parameter language model built for mathematics ^[1]. Wikipedia records the DeepSeek-Math release in April 2024 ^[2]. Alongside the model, the paper introduced a new reinforcement-learning algorithm, Group Relative Policy Optimization (GRPO) ^[1].

How the model was built

The model was not trained from scratch. The team continued pre-training DeepSeek-Coder-Base-v1.5 7B on 120 billion math-related tokens mined from Common Crawl, together with natural-language and code data ^[1]. The starting point was a code model, and the added corpus was dominated by mathematical text pulled out of a general web crawl, so the recipe combines three ingredients: mathematics, ordinary language and code.

Results

Without external toolkits — no calculator or computer-algebra system — and without voting over multiple samples, DeepSeekMath 7B scored 51.7% on the MATH benchmark ^[1]. With self-consistency over 64 samples, that is, generating 64 answers per problem and taking the most frequent one, the score rose to 60.9% ^[1]. The authors attributed the gains to two things: a data-selection pipeline for public web data, and the new reinforcement-learning algorithm ^[1].

GRPO: dropping the critic

The second contribution is the algorithm. PPO depends on a learned value function, typically a model of comparable size to the policy model ^[1]. GRPO removes that critic model entirely and instead estimates the baseline from the average reward of a group of sampled outputs produced for the same question ^[1].

The idea is easy to state. For a given problem the model samples several answers; each answer is scored; each score is then compared not against a learned prediction but against the average of the group. Answers above the group average are reinforced, answers below it are suppressed. Because the reference point comes from the group itself, no separate value network has to be trained or kept in memory, and that is where the saving comes from. The paper describes GRPO as a variant of PPO that improves mathematical reasoning while reducing PPO's memory usage ^[1].

Why it matters

Two contributions can be separated. The first is a data recipe: a pipeline that selects mathematical text out of public web data, which the authors credit as one source of the gains ^[1]. The second is an algorithm that removes an entire model from the reinforcement-learning loop. Memory is often the constraint that decides how large a model can be trained, or how many samples can be drawn per question, so an algorithm that improves reasoning without a critic model is useful well beyond mathematics.