DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Chain-of-thought prompting showed that models reason better when they write out steps. OpenAI's o1 showed that training a model to produce long internal reasoning helps much more, but not how it was done. The usual assumption was that you need many human-written examples of good reasoning.

DeepSeek-AI tested the opposite. For DeepSeek-R1-Zero, they took their DeepSeek-V3 base model and ran large-scale reinforcement learning with no supervised fine-tuning at all. The only instruction was a template: think inside <think> tags, then answer. The only rewards were rule-based: is the final answer correct (a math answer matches, code passes tests), and is the format right. The algorithm, GRPO, samples a group of answers per question and rewards each one relative to the group's average, which avoids training a separate value model.

Over about 10,400 training steps, R1-Zero's accuracy on AIME 2024 math problems rose from 15.6% to 77.9%, and to 86.7% when voting over 16 samples. Its answers grew longer on their own, from hundreds to thousands of tokens, and it started checking its work, trying alternatives and writing "wait" mid-solution. No one showed it how.

R1-Zero was hard to read and mixed English and Chinese, and it was weak outside reasoning. So DeepSeek-R1 added stages: a small "cold start" fine-tune on thousands of readable reasoning examples, reasoning RL with a reward for staying in one language, a large supervised fine-tune on 800,000 rejection-sampled examples covering reasoning and general tasks, and a final RL stage for helpfulness and harmlessness. The result matched OpenAI o1-1217 on math (79.8% AIME 2024, 97.3% MATH-500) and was close on code. Fine-tuning small open models on R1's outputs produced distilled models that beat GPT-4o on math, down to 1.5B parameters.

R1 opened reasoning models to everyone: open weights, a published recipe, and distilled versions small enough to self-host. For customer work it raises three practical questions.

When to use a reasoning model. They help on problems with a definite answer that needs several steps: math, code, logic, multi-constraint planning. They cost more tokens and time, and the paper notes R1 still "overthinks" simple questions. Route easy requests to a normal model.

How to prompt them. The paper found few-shot prompting consistently hurt R1; describe the problem and the output format directly instead. That reverses a habit many teams carried over from earlier models.

What it says about customization. The reward signal is the whole method: correct-or-not checks, not human judgments. If a customer's task has a reliable checker (tests, totals, schema validity, a policy rule), reinforcement learning against it is a real option, and the authors warn that model-based rewards were "hacked" during training. Where no checker exists, they fell back to human-annotated supervised data.

Distillation is the cheap path. For most customers, the practical route is a distilled model, or fine-tuning a small model on a strong reasoning model's outputs for their domain. In the paper, distilling R1 into a 32B model beat running large-scale RL on that same 32B model.

Know the gaps. At release R1 was weak on structured output and could not use tools during reasoning. Check both before putting it in an agent.