DeepSeekMath samples 64 responses per question, a teaching implementation defaults to 8, and the latest paper claims even 2 is enough — a 32x spread on the same knob. G is GRPO’s only source of a baseline and its biggest compute lever, so what does it actually buy? This post unpacks G’s three roles: the sample size of the control variate, the coverage of groups with effective gradients, and the quality of the std-normalization denominator, plus why dynamic sampling can replace a larger G with filtering.

This post has a companion interactive report (in Chinese), with a zero-gradient-group probability calculator, a dual-curve baseline chart, and a side-by-side comparison of the three camps’ choices.

A Knob With a 32x Spread

On every training step, GRPO samples a group of GG responses for each prompt. How large this GG should be is a question on which different camps give wildly different answers:

  • DeepSeekMath (the original GRPO paper, arXiv:2402.03300 §4.2): 64. With a batch size of 1024, that means a single step covers only 16 prompts;
  • The RLHF Book’s companion teaching code (code/policy_gradients/configs/): 8, the default across the whole lineup, from GRPO to GSPO and CISPO;
  • The October 2025 paper “It Takes Two: Your GRPO Is Secretly DPO” (arXiv:2510.00977): 2 is enough, and the paper argues specifically that the “large groups are necessary” view is a misreading of why GRPO succeeds.

A 32x gap. Worse, GG is two things at once: it is GRPO’s only source of a baseline (there is no critic; the group mean is the baseline), and it is the biggest multiplier on training cost (rollouts are the main compute expense in RL training, and doubling GG doubles that cost). One knob governs both signal quality and the compute bill, yet people have gotten everything from 2 to 64 to work — which suggests that the question “what does GG actually buy?” is itself worth taking apart.

Role One: The Baseline’s Sample Size

Having dropped the critic, GRPO uses the group mean as its baseline: A^i=ri−mean({rj}j=1G)\hat{A}_i = r_i - \text{mean}(\{r_j\}_{j=1}^G) (setting aside the division by std for now). Subtracting a baseline is the standard variance-reduction technique for policy gradients, and the validity condition the textbook (RLHF Book, Chapter 6) gives is that the baseline must not depend on the current action, because Ea∼π[b(s)∇θlog⁡πθ(a∣s)]=0\mathbb{E}_{a\sim\pi}[b(s)\nabla_\theta\log\pi_\theta(a|s)] = 0 holds for any b(s)b(s) that depends only on the state — the baseline leaves the expected gradient unchanged and only suppresses variance.

There is a detail here that few people mention: GRPO’s mean includes the very response being evaluated, so strictly speaking the baseline depends on the action, and the unbiasedness condition is not strictly satisfied. Compare RLOO (REINFORCE Leave-One-Out, same chapter of rlhf-book): its baseline is the mean of the other K−1K-1 responses, deliberately excluding the response itself, to “keep the gradient estimator fully unbiased.” The relationship between the two advantages can be written out directly:

A^iRLOO=ri−1G−1∑j≠irj=GG−1(ri−1G∑jrj)=GG−1A^iGRPO(no std)\hat{A}_i^{\text{RLOO}} = r_i - \frac{1}{G-1}\sum_{j\neq i}r_j = \frac{G}{G-1}\left(r_i - \frac{1}{G}\sum_{j}r_j\right) = \frac{G}{G-1}\hat{A}_i^{\text{GRPO(no std)}}

The directions are identical; the two differ only by a scale factor of G/(G−1)G/(G-1). At G=64G=64 the factor is 1.016, which hardly matters; at G=8G=8 it is 1.14; at G=2G=2 it is 2 — the GRPO-form advantage is only half as large as the leave-one-out form. This bias is not directional, just a rescaling that grows as GG shrinks, and in practice its effect can be absorbed by the learning rate (this is an inference, not an experimental finding from any paper), but it explains why every property of small GG gets amplified.

The second entry in the ledger is the baseline’s estimation noise. The group mean is an estimate of the true expectation E[r]\mathbb{E}[r] from GG samples, with noise of magnitude roughly σr/G\sigma_r/\sqrt{G} (σr\sigma_r is the standard deviation of a single response’s reward). When GG is small, this noise leaks straight into every response’s advantage: a correct response can end up with a negative advantage because “everyone else in its group happened to be strong too.” The probability that baseline noise randomly flips the gradient direction rises as GG decreases — this is the standard “large GG reduces variance” argument, and also what 2510.00977 sets out to challenge; more on that later.

Role Two: Coverage of Groups With Effective Gradients

Rewards in RLVR are binary (right or wrong; DAPO uses +1/−1, DeepSeekMath uses 1/0, and the two are equivalent). If a group’s GG responses are all right or all wrong, there is no reward variance within the group and every advantage is zero — those GG rollouts were run for nothing and contribute no gradient at all.

The probability that a group is uniform can be computed directly (what follows is my own calculation): let the model’s per-sample accuracy on the question be pp, with samples independent; then

P(zero-gradient group)=pG+(1−p)GP(\text{zero-gradient group}) = p^G + (1-p)^G

Plugging in numbers to see the trend (pp is the per-sample accuracy; the table gives the probability that the group has zero gradient):

ppG=8G=8G=64G=64
0.50.8%≈0
0.816.8%≈0
0.943.0%0.1%
0.9566.3%3.8%
0.9992.3%52.6%

How to read it: for medium-difficulty questions (pp in the 0.3–0.7 range), even a small GG is basically safe; what really consumes GG is questions at the difficulty boundary — at p=0.95p=0.95, with G=8G=8 two-thirds of the groups are all-correct and gradient-free, and only G=64G=64 pushes that down to 4%. And the training trajectory pushes questions precisely toward the boundaries: for questions the model has learned, p→1p \to 1; for questions it can’t learn, p→0p \to 0; and the later in training, the fewer questions remain in the “gradient-bearing range.” This is one source of the “training efficiency and stability” problems that the DAPO paper describes in its motivation.

Faced with the same table, there are two remedies. One is to increase GG and brute-force pGp^G down — with costs growing multiplicatively. The other is DAPO’s Dynamic Sampling: instead of increasing GG, simply rule that zero-gradient groups are not accepted. DAPO’s objective includes a hard constraint (the s.t. condition of Equation 8 in the paper):

0<∣{oi∣is_equivalent(a,oi)}∣<G0 < \left|\{o_i \mid \texttt{is\_equivalent}(a, o_i)\}\right| < G

That is, the number of correct responses in a group must lie strictly between 0 and GG — all-correct and all-wrong groups are filtered out and replaced through oversampling, guaranteeing that every prompt in the batch has an effective gradient. For the same effective-group coverage, the price of filtering is additive oversampling rather than a multiplicative increase in GG.

But there is another layer of coupling between filtering and GG that is easy to miss: the filtering decision is itself based on an estimate from GG samples. At G=4G=4, “all 4 wrong” does not mean the model can’t solve the question — a question with p=0.3p=0.3 has a 24% chance of getting all 4 wrong and being mistakenly thrown on the scrap heap. Pairing a small GG with dynamic sampling will misjudge a whole set of medium-difficulty questions that “actually carry gradient value” as dead groups; the misjudgment rate falls exponentially with GG ((1−p)G (1-p)^G , the same table). So dynamic sampling does not eliminate GG’s coverage problem; it merely converts it from “zero gradient” into “mistaken discards” (this point is an inference; the DAPO paper does not discuss the misjudgment rate).

Role Three: The Quality of the std-Normalization Denominator

GRPO’s advantage is also divided by the group std: A^i=(ri−mean)/std\hat{A}_i = (r_i - \text{mean})/\text{std}. This std is likewise an estimate from GG samples and is very noisy when GG is small; for a uniform group the std is exactly 0, so implementations either add an eps or skip the group (setting a divide-by-zero trap).

Dr.GRPO’s critique of the std (see the dedicated post) — that when the std of all-correct or all-wrong groups approaches zero, the advantage is inflated to unreasonable magnitudes — is at its worst with small GG: when GG is small, the std is more likely to land on a small value, and extreme advantages show up frequently. Extreme advantages in turn push up how far the ratio moves, indirectly raising the clip fraction (this chain is an inference: abnormally large A^\hat{A} → πθ\pi_\theta moves more within a single step → wtw_t hits the clipping bound sooner). In other words, part of the claim that “large GG stabilizes training” is really not about the baseline but about the std — two ledgers that the literature often lumps together.

One more theoretical result worth knowing (arXiv:2502.18548, “What is the Alignment Objective of GRPO?”): the authors give a framework characterizing GRPO’s stationary policies and point out that GRPO’s way of aggregating preferences differs fundamentally from the logarithmic pooling of standard RLHF — GG affects not only the variance of the estimate but also the fixed point of the normalized objective itself. This is analytical work, and the conditions under which its conclusions hold (stationarity, infinitely many steps) are some distance from engineering practice, but it is a reminder of one thing: understanding GG purely as a “sampling budget” is incomplete.

Counterpoint: Are 2 Responses Enough?

The argument of 2510.00977 is worth restating on its own terms: GRPO’s effectiveness comes from an implicit contrastive objective — comparing just two responses already yields a learning signal, and the group mean is merely a control variate for variance reduction that does not need to be a precise statistical estimate, so 2-GRPO with G=2G=2 also works and, structurally, even ends up connecting to DPO. Their theoretical analysis and experiments (at small to medium scale) both support this conclusion.

Does this contradict DeepSeekMath’s G=64G=64? Not necessarily. Of the things a large GG buys, baseline precision is only one, and possibly the least important. The others are coverage of boundary-difficulty questions (the second role), the reliability of the filtering decision (the companion to dynamic sampling), and exploration coverage — a group needs at least one correct response for the model to have a “good sample” to learn from; for a hard question with p=0.1p=0.1, there is an 81% chance at G=2G=2 that the group contains no positive sample, versus only 0.1% at G=64G=64. These gains depend strongly on model scale and on the task difficulty distribution: for a small model grinding through GSM8K-level problems, the pp distribution is concentrated in the middle range, so a small GG costs little; for a large model attacking AIME-level problems, pp is pinned around 0.1, every positive sample is precious, and GG is the exploration budget itself. Whether the 2-GRPO conclusion extends to large models in long-CoT settings is something nobody can answer yet (the paper’s experiments are limited in scale; this is a boundary, not a criticism).

The Cost Ledger, and My Take

First, the accounting: rollouts per training step = number of prompts × GG. DeepSeekMath’s configuration (batch 1024, G=64G=64) gets through only 16 prompts per step — prompt diversity and response diversity are traded against each other on the same budget sheet. One more configuration detail deserves attention: DeepSeekMath performs only one policy update per exploration stage (strictly on-policy, §4.2), so the ratio is always 1 and clipping never triggers, and GG’s entire effect shows up in the baseline and in coverage; whereas with DAPO/GSPO-style multi-mini-batch updates, GG also indirectly affects the clip fraction through std noise. The same GG does not have the same marginal return in the two training regimes.

My take: there is no universally optimal GG, because it spends money on three budget sheets at once — baseline precision (σr/G\sigma_r/\sqrt{G}, with the fastest-diminishing returns), effective-group coverage (pGp^G, with the most durable returns at boundary difficulty), and the exploration budget (the probability that a group contains a positive sample, almost linear on hard questions). The practical default (8–16) is the balance point under “a medium-difficulty distribution + dynamic sampling as a safety net”; 64 was the naive choice of the DeepSeekMath era, before dynamic sampling and before any clip modifications; 2 is the theoretical limit from the “contrastive objective” perspective, which in engineering terms nobody has yet dared to use at production scale. Before picking GG, first ask yourself which of the three sheets is the bottleneck: if the questions are too hard (the exploration budget is tight), increase GG or add dynamic sampling; if the questions are too varied (prompt diversity is tight), give the budget to the number of prompts; if training is unstable, check the std and the clip fraction first, and don’t rush to increase GG.

The first four posts in this series happen to form a single thread: Clip-Higher fixes the token-level clipping bounds, GSPO changes the unit of the ratio, TIS plugs the numerical gap between training and inference, and this post goes back to the source of the signal — how to allocate the sampling budget. Every “self-evident” default configuration in GRPO, once taken apart, turns out to have a ledger of its own.

Open Questions

  • There is no public systematic ablation of the optimal coupling between GG, the learning rate, and the clip ε\varepsilon; whether small GG + a large learning rate can match the signal-to-noise ratio of a large GG is backed only by scattered folk wisdom;
  • No paper has quantified the actual impact on data efficiency of dynamic sampling’s “mistaken discard” rate ((1−p)G(1-p)^G) on medium-difficulty questions;
  • Whether 2-GRPO holds for large models, long CoT, and boundary-difficulty tasks is one of the most worthwhile public replications to attempt right now.

References