DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

The problem

Reasoning models are language models that work through a problem step by step before answering. Building on the chain-of-thought idea, the DeepSeek team asked whether this ability could be developed mainly through reinforcement learning, rewarding good answers and letting the model find better ways to reason.^[1]

Reinforcement learning is a natural fit for this question because it does not need a written example of every correct reasoning path, only a way to score the outcome.

Publication

DeepSeek released DeepSeek-R1 on 20 January 2025, with open weights.^[2] The technical report describing its training was submitted to arXiv on 22 January 2025.^[1]

How it worked

The team first built DeepSeek-R1-Zero. It started from the DeepSeek-V3 base model and was trained with large-scale reinforcement learning alone, without a supervised fine-tuning stage first. Through this training it learned to produce long chains of reasoning and to check and revise its own work.^[1]

Training with only reinforcement learning is a demanding test of whether this behaviour can emerge from the reward alone.

In reinforcement learning, a model's outputs are scored and it adjusts its behaviour to earn higher scores. Where possible, the rewards were rule-based: a math answer earned reward for being correct, and a code solution for passing tests. Format rewards were also used.^[1]

A problem with a checkable answer gives a clear signal without needing a person to judge each response, which is why rule-based rewards were practical here.

R1-Zero had problems with readability, and its responses sometimes mixed languages. DeepSeek-R1 addressed this by adding a small amount of cold-start supervised data, meaning an initial set of examples used before reinforcement learning, along with further training stages.^[1]

Results

The technical report states that DeepSeek-R1 performs comparably to OpenAI's o1 on math, code and reasoning benchmarks. Benchmarks are standardised tests used to compare models.^[1]

A benchmark result shows how a model performed on a particular set of tasks. It does not guarantee the same level of performance on every problem a user might bring.

The weights were released under the MIT license, together with smaller distilled models based on Qwen and Llama.^[2] Distillation means training a smaller model to imitate the outputs of a larger one.

Why it matters

The report described its R1-Zero result as the first open research to show that reasoning in large language models can be encouraged purely through reinforcement learning.^[1] The release drew wide attention to the Chinese lab: on 27 January 2025, concerns about competition from DeepSeek helped wipe almost $600 billion off Nvidia's market value in a single day.^[3]

For readers, the main lesson is that step-by-step reasoning can be developed through reinforcement learning, not only through examples of reasoning written by people.

What followed

Because the training process is documented in the report, other researchers can study it directly and compare it with their own methods.

The open weights mean that the released models can be run, tested and compared by anyone with the hardware to do so.