GRPO objective

Background

Group Relative Policy Optimization (GRPO) is the RL objective behind models like DeepSeek-R1. It is a PPO variant that drops the value network: instead of a learned baseline, it estimates advantages by comparing the rewards of a group of sampled responses to the same prompt. It keeps PPO's clipped ratio for stability and adds a KL penalty pulling the policy toward a reference model.

Problem statement

Implement grpo_objective(rhos, A, pi_theta_old, pi_theta_ref, epsilon=0.2, beta=0.01):

$$L = \underbrace{\frac{1}{G}\sum_i \min\!\big(\rho_i A_i,\ \operatorname{clip}(\rho_i, 1-\epsilon, 1+\epsilon)A_i\big)}_{\text{clipped surrogate}} \;-\; \beta\, D_{KL}(\pi_\theta \,\|\, \pi_{\text{ref}})$$

Recover $\pi_\theta \propto \rho \cdot \pi_{\text{old}}$, normalize both $\pi_\theta$ and $\pi_{\text{ref}}$ to sum to 1, and use $D_{KL} = \sum \pi_\theta \log(\pi_\theta / \pi_{\text{ref}} + 10^{-10})$.

Input

• rhos — likelihood ratios $\rho_i = \pi_\theta(o_i)/\pi_{\text{old}}(o_i)$, length G.
• A — advantage estimates, length G.
• pi_theta_old, pi_theta_ref — old-policy and reference probabilities, length G.
• epsilon, beta — clip range and KL coefficient.

Raise ValueError if the lengths differ.

Output

A scalar float: the GRPO objective.

Examples

Example 1

grpo_objective([1.2, 0.8, 1.1], [1.0, 1.0, 1.0], [0.9, 1.1, 1.0], [1.0, 0.5, 1.5], 0.2, 0.01)
-> 1.032749

Explanation: ratios are within $1\pm0.2$ so nothing clips; the average min term is $3.1/3=1.0333$. Subtracting the small KL penalty ($\beta\cdot 0.0584$) gives $\approx 1.0327$.

Constraints

• Clip $\rho$ to $[1-\epsilon, 1+\epsilon]$ and take the elementwise min with the unclipped term, then average.
• Build $\pi_\theta = \rho \cdot \pi_{\text{old}}$, then normalize $\pi_\theta$ and $\pi_{\text{ref}}$ to probability distributions before the KL.
• Add $10^{-10}$ inside the log for numerical stability.

Notes

• Dropping the value network (PPO's critic) is GRPO's main simplification — group-relative advantages replace it, which is cheaper for LLM-scale RL.
• The KL term keeps the fine-tuned policy close to the reference, preventing reward-hacking drift.