← All refreshers Paper capsule · 8 min read

Paper capsule: DeepSeekMath, where GRPO comes from

Shao, Wang, Zhu, et al. (DeepSeek-AI), 2024 · arXiv 2402.03300

What Shao et al. (2024) built, why they invented GRPO instead of using PPO, and what they found about what RL actually improves.

Skip if You've read DeepSeekMath (arXiv 2402.03300) and know why GRPO drops the value network and where its KL term lives.

The paper in brief

DeepSeekMath (Zhihong Shao and colleagues at DeepSeek-AI, February 2024, arXiv 2402.03300) set out to push a small open model as far as possible on mathematical reasoning. It made three contributions, and the third is why every RL paper since cites it:

  1. Data. They mined about 120 billion tokens of math-related text from Common Crawl. Starting from a known-good math dataset as seed examples, they trained a fast text classifier to recognize math pages, harvested more pages with it, refined the classifier, and repeated over several rounds.
  2. A 7B model that punches above its size. They continued pretraining from a code model (DeepSeek-Coder-Base-v1.5 7B), reporting that starting from code helps mathematical reasoning. They then instruction-tuned it and trained it further with RL. The final 7B model reached 51.7% on the competition-level MATH benchmark without tools or voting, and 60.9% with majority voting over 64 samples, close to the much larger frontier models of the time.
  3. GRPO, Group Relative Policy Optimization. This is a leaner variant of PPO, built for the case where the reward comes only at the end of a response.

Why not just use PPO?

PPO (see the policy-gradients refresher) needs a value network, a critic that predicts the expected reward from every token position. That causes two problems for language models:

  • Cost. The critic is typically about as large as the policy, so it roughly doubles the memory and compute of training.
  • A hard learning problem. When the reward arrives only at the final token (right answer or wrong answer), the critic has to guess the eventual outcome from every intermediate prefix. That’s a difficult regression problem, and a poor critic produces noisy advantages.

GRPO’s fix: for each question, sample a group of GG answers, and use the group’s own average reward as the baseline. No critic is needed.

The objective

For a question qq with sampled outputs o1,…,oGo_1, \dots, o_G, GRPO maximizes

J(θ)=E[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡(ρi,tA^i,t,clip(ρi,t,1−ϵ,1+ϵ)A^i,t)−βDKL[πθ∥πref])]\mathcal{J}(\theta) = \mathbb{E}\left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Big( \min\big(\rho_{i,t} \hat A_{i,t},\ \mathrm{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon)\, \hat A_{i,t}\big) - \beta\, \mathbb{D}_{\text{KL}}\big[\pi_\theta \,\|\, \pi_{\text{ref}}\big] \Big) \right]

where ρi,t\rho_{i,t} is the probability ratio between the new and old policy for token tt of output ii. Three details are worth knowing:

  • The advantage. With outcome rewards, every token of output ii gets A^i,t=(ri−mean(r))/std(r)\hat A_{i,t} = (r_i - \mathrm{mean}(\mathbf{r})) / \mathrm{std}(\mathbf{r}).
  • The KL term goes in the loss, not the reward. Classic RLHF subtracts a KL penalty from the reward at each token; GRPO adds it directly to the objective. It uses an estimator that is unbiased and always non-negative:
DKL=πrefπθ−log⁡πrefπθ−1\mathbb{D}_{\text{KL}} = \frac{\pi_{\text{ref}}}{\pi_\theta} - \log\frac{\pi_{\text{ref}}}{\pi_\theta} - 1
  • Per-output length normalization. The factor 1/∣oi∣1/|o_i| averages over each output’s own tokens. That choice matters again once an output is split into many segments, as in the CLM paper’s stepwise variant.

The paper also describes a process-supervision version, where a reward model scores each reasoning step and a token’s advantage sums the normalized rewards of the steps after it. And it describes iterative RL, where the reward model is periodically retrained on the policy’s newest outputs.

The finding people forget

DeepSeekMath also proposes a unified view of training methods. SFT, rejection-sampling fine-tuning, DPO, PPO, and GRPO all become variations in three ingredients: where the data comes from, how it’s scored, and how strongly each sample pushes the gradient.

Their most quoted observation: RL improved majority-vote accuracy over many samples, but not the chance that any of K samples is correct. Their reading is that RL mostly makes the model more reliable at producing answers it could already produce, sharpening the output distribution rather than adding fundamentally new abilities.

Why it matters for later papers

GRPO became the default RL algorithm for open reasoning models. DeepSeek-R1 used it in early 2025, and most agent-RL papers since build on it with add-ons such as dynamic sampling and asymmetric clipping. When a paper says “we use GRPO,” it almost always means this objective plus a few such tweaks, so the first questions to ask are which tweaks, and what counts as one output for the advantage and the 1/∣oi∣1/|o_i| normalization.