← All refreshers Refresher · 12 min read

Policy gradients to GRPO

From REINFORCE to group-relative advantages, and how it extends to multi-call agent trajectories.

Skip if You can derive REINFORCE, explain why a baseline keeps it unbiased, and know what GRPO's group baseline does.

The setup

Treat the language model as a policy πθ(y∣x)\pi_\theta(y \mid x): given a prompt xx, it samples a response yy. A reward function R(x,y)R(x, y) scores the response; for deep research, that’s 1 if the final answer is judged correct and 0 otherwise. We want to maximize the expected reward:

J(θ)=Ex∼D,y∼πθ(⋅∣x)[R(x,y)]J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi_\theta(\cdot \mid x)}\big[R(x, y)\big]

The difficulty is that yy is sampled, so we can’t backpropagate through the sampling step.

The log-derivative trick (REINFORCE)

For a fixed prompt, write the expectation as a sum and move the gradient inside:

∇θJ=∑y∇θπθ(y)R(y)=∑yπθ(y)∇θlog⁡πθ(y)R(y)=Ey[R(y)∇θlog⁡πθ(y)]\nabla_\theta J = \sum_y \nabla_\theta \pi_\theta(y)\, R(y) = \sum_y \pi_\theta(y)\, \nabla_\theta \log \pi_\theta(y)\, R(y) = \mathbb{E}_y\big[R(y)\, \nabla_\theta \log \pi_\theta(y)\big]

The middle step uses ∇π=π∇log⁡π\nabla \pi = \pi \, \nabla \log \pi. Now the gradient is an expectation we can estimate by sampling: generate responses, score them, and push up the log-probability of each response in proportion to its reward. For a language model, the log-probability of a response is a sum over its tokens:

log⁡πθ(y∣x)=∑klog⁡πθ(yk∣x,y1:k−1)\log \pi_\theta(y \mid x) = \sum_{k} \log \pi_\theta\big(y_k \mid x,\, y_{1:k-1}\big)

so every token in a response receives the same scalar weight.

Baselines: same expectation, less variance

Raw rewards make a noisy signal. For example, if every sample scores 1, every response gets pushed up, even though nothing distinguishes them. The fix is to subtract a baseline b(x)b(x) that doesn’t depend on yy. It leaves the gradient unbiased because

Ey[∇θlog⁡πθ(y)]=∑y∇θπθ(y)=∇θ∑yπθ(y)=∇θ1=0.\mathbb{E}_y\big[\nabla_\theta \log \pi_\theta(y)\big] = \sum_y \nabla_\theta \pi_\theta(y) = \nabla_\theta \sum_y \pi_\theta(y) = \nabla_\theta 1 = 0 .

The weight becomes the advantage A=R−b(x)A = R - b(x), which measures how much better this response was than expected.

PPO: reusing samples safely

Sampling is expensive, so PPO takes several gradient steps on each batch of samples. It corrects for the policy drifting away from the one that generated the samples with an importance ratio ρ=πθ(yk∣⋅)/πold(yk∣⋅)\rho = \pi_\theta(y_k \mid \cdot) / \pi_{\text{old}}(y_k \mid \cdot), and clips that ratio so no single update moves the policy too far:

Lclip=E[min⁡(ρA,clip(ρ,1−ϵlow,1+ϵhigh)A)]\mathcal{L}^{\text{clip}} = \mathbb{E}\Big[\min\big(\rho A,\; \mathrm{clip}(\rho,\, 1-\epsilon_{\text{low}},\, 1+\epsilon_{\text{high}})\, A\big)\Big]

Classic PPO estimates AA with a learned value network (a critic), which is a second model roughly as large as the policy.

GRPO: the group is the baseline

GRPO (Shao et al., 2024, introduced in DeepSeekMath) drops the critic. For each prompt, it samples a group of GG responses with rewards r1,…,rGr_1, \dots, r_G and normalizes within the group:

Ai=ri−mean(r1,…,rG)std(r1,…,rG)A_i = \frac{r_i - \mathrm{mean}(r_1, \dots, r_G)}{\mathrm{std}(r_1, \dots, r_G)}

Every token of response ii gets AiA_i. A KL penalty keeps the policy close to a reference model.

Binary rewards make this concrete. If a fraction p^\hat p of the group succeeds, the mean is p^\hat p and the standard deviation is p^(1−p^)\sqrt{\hat p (1 - \hat p)}, so

Asuccess=1−p^p^,Afailure=−p^1−p^.A_{\text{success}} = \sqrt{\frac{1 - \hat p}{\hat p}}, \qquad A_{\text{failure}} = -\sqrt{\frac{\hat p}{1 - \hat p}} .

On a hard prompt where only 2 of 32 attempts succeed, each success gets an advantage of about 3.93.9, a strong push. On an easy prompt where 30 of 32 succeed, each success gets only about 0.260.26.

Three practical add-ons, all of which the CLM paper uses:

  1. Dynamic sampling. If every rollout in a group succeeds or every one fails, the standard deviation is zero and the group carries no signal, so these groups are dropped.
  2. Clip-higher. An asymmetric clip range, [1−0.2,1+0.28][1 - 0.2,\; 1 + 0.28] in the paper, lets rare good tokens gain probability faster than the default symmetric range would.
  3. Truncated importance sampling. Rollouts come from a fast inference engine (SGLang or vLLM) whose token probabilities differ slightly from the trainer’s. Capping the correction ratio keeps that mismatch from blowing up the gradient.

From one response to an agent trajectory

An agent doesn’t produce one response; it makes many model calls. Call jj sees an input cjc_j (the context at that moment) and produces an output yjy_j. Between calls, the environment (tools, files) responds, but the environment doesn’t depend on θ\theta. So the trajectory’s log-probability still decomposes into per-call terms:

∇θlog⁡pθ(τ)=∑j∇θlog⁡πθ(yj∣cj)\nabla_\theta \log p_\theta(\tau) = \sum_j \nabla_\theta \log \pi_\theta\big(y_j \mid c_j\big)

In an append-only agent, each cjc_j is a prefix of the final transcript, so you can train on the final transcript and simply mask out the tool outputs. When the model edits its own context, that shortcut breaks: the final transcript no longer contains what earlier calls actually saw. The fix is bookkeeping, not new math. Store every (cj,yj)(c_j, y_j) pair as its own segment, and give each segment the trajectory’s group-relative advantage. That’s stepwise GRPO, which we dissect in §4.2.