DeepSeekMath samples 64 responses per question, a teaching implementation defaults to 8, and the latest paper claims even 2 is enough — a 32x spread on the same knob. G is GRPO’s only source of a baseline and its biggest compute lever, so what does it actually buy? This post unpacks G’s three roles: the sample size of the control variate, the coverage of groups with effective gradients, and the quality of the std-normalization denominator, plus why dynamic sampling can replace a larger G with filtering.
This post has a companion interactive report (in Chinese), with a zero-gradient-group probability calculator, a dual-curve baseline chart, and a side-by-side comparison of the three camps’ choices.
A Knob With a 32x Spread
On every training step, GRPO samples a group of responses for each prompt. How large this should be is a question on which different camps give wildly different answers:
- DeepSeekMath (the original GRPO paper, arXiv:2402.03300 §4.2): 64. With a batch size of 1024, that means a single step covers only 16 prompts;
- The RLHF Book’s companion teaching code (
code/policy_gradients/configs/): 8, the default across the whole lineup, from GRPO to GSPO and CISPO; - The October 2025 paper “It Takes Two: Your GRPO Is Secretly DPO” (arXiv:2510.00977): 2 is enough, and the paper argues specifically that the “large groups are necessary” view is a misreading of why GRPO succeeds.
A 32x gap. Worse, is two things at once: it is GRPO’s only source of a baseline (there is no critic; the group mean is the baseline), and it is the biggest multiplier on training cost (rollouts are the main compute expense in RL training, and doubling doubles that cost). One knob governs both signal quality and the compute bill, yet people have gotten everything from 2 to 64 to work — which suggests that the question “what does actually buy?” is itself worth taking apart.
Role One: The Baseline’s Sample Size
Having dropped the critic, GRPO uses the group mean as its baseline: (setting aside the division by std for now). Subtracting a baseline is the standard variance-reduction technique for policy gradients, and the validity condition the textbook (RLHF Book, Chapter 6) gives is that the baseline must not depend on the current action, because holds for any that depends only on the state — the baseline leaves the expected gradient unchanged and only suppresses variance.
There is a detail here that few people mention: GRPO’s mean includes the very response being evaluated, so strictly speaking the baseline depends on the action, and the unbiasedness condition is not strictly satisfied. Compare RLOO (REINFORCE Leave-One-Out, same chapter of rlhf-book): its baseline is the mean of the other responses, deliberately excluding the response itself, to “keep the gradient estimator fully unbiased.” The relationship between the two advantages can be written out directly:
The directions are identical; the two differ only by a scale factor of . At the factor is 1.016, which hardly matters; at it is 1.14; at it is 2 — the GRPO-form advantage is only half as large as the leave-one-out form. This bias is not directional, just a rescaling that grows as shrinks, and in practice its effect can be absorbed by the learning rate (this is an inference, not an experimental finding from any paper), but it explains why every property of small gets amplified.
The second entry in the ledger is the baseline’s estimation noise. The group mean is an estimate of the true expectation from samples, with noise of magnitude roughly ( is the standard deviation of a single response’s reward). When is small, this noise leaks straight into every response’s advantage: a correct response can end up with a negative advantage because “everyone else in its group happened to be strong too.” The probability that baseline noise randomly flips the gradient direction rises as decreases — this is the standard “large reduces variance” argument, and also what 2510.00977 sets out to challenge; more on that later.
Role Two: Coverage of Groups With Effective Gradients
Rewards in RLVR are binary (right or wrong; DAPO uses +1/−1, DeepSeekMath uses 1/0, and the two are equivalent). If a group’s responses are all right or all wrong, there is no reward variance within the group and every advantage is zero — those rollouts were run for nothing and contribute no gradient at all.
The probability that a group is uniform can be computed directly (what follows is my own calculation): let the model’s per-sample accuracy on the question be , with samples independent; then
Plugging in numbers to see the trend ( is the per-sample accuracy; the table gives the probability that the group has zero gradient):
| 0.5 | 0.8% | ≈0 |
| 0.8 | 16.8% | ≈0 |
| 0.9 | 43.0% | 0.1% |
| 0.95 | 66.3% | 3.8% |
| 0.99 | 92.3% | 52.6% |
How to read it: for medium-difficulty questions ( in the 0.3–0.7 range), even a small is basically safe; what really consumes is questions at the difficulty boundary — at , with two-thirds of the groups are all-correct and gradient-free, and only pushes that down to 4%. And the training trajectory pushes questions precisely toward the boundaries: for questions the model has learned, ; for questions it can’t learn, ; and the later in training, the fewer questions remain in the “gradient-bearing range.” This is one source of the “training efficiency and stability” problems that the DAPO paper describes in its motivation.
Faced with the same table, there are two remedies. One is to increase and brute-force down — with costs growing multiplicatively. The other is DAPO’s Dynamic Sampling: instead of increasing , simply rule that zero-gradient groups are not accepted. DAPO’s objective includes a hard constraint (the s.t. condition of Equation 8 in the paper):
That is, the number of correct responses in a group must lie strictly between 0 and — all-correct and all-wrong groups are filtered out and replaced through oversampling, guaranteeing that every prompt in the batch has an effective gradient. For the same effective-group coverage, the price of filtering is additive oversampling rather than a multiplicative increase in .
But there is another layer of coupling between filtering and that is easy to miss: the filtering decision is itself based on an estimate from samples. At , “all 4 wrong” does not mean the model can’t solve the question — a question with has a 24% chance of getting all 4 wrong and being mistakenly thrown on the scrap heap. Pairing a small with dynamic sampling will misjudge a whole set of medium-difficulty questions that “actually carry gradient value” as dead groups; the misjudgment rate falls exponentially with (, the same table). So dynamic sampling does not eliminate ’s coverage problem; it merely converts it from “zero gradient” into “mistaken discards” (this point is an inference; the DAPO paper does not discuss the misjudgment rate).
Role Three: The Quality of the std-Normalization Denominator
GRPO’s advantage is also divided by the group std: . This std is likewise an estimate from samples and is very noisy when is small; for a uniform group the std is exactly 0, so implementations either add an eps or skip the group (setting a divide-by-zero trap).
Dr.GRPO’s critique of the std (see the dedicated post) — that when the std of all-correct or all-wrong groups approaches zero, the advantage is inflated to unreasonable magnitudes — is at its worst with small : when is small, the std is more likely to land on a small value, and extreme advantages show up frequently. Extreme advantages in turn push up how far the ratio moves, indirectly raising the clip fraction (this chain is an inference: abnormally large → moves more within a single step → hits the clipping bound sooner). In other words, part of the claim that “large stabilizes training” is really not about the baseline but about the std — two ledgers that the literature often lumps together.
One more theoretical result worth knowing (arXiv:2502.18548, “What is the Alignment Objective of GRPO?”): the authors give a framework characterizing GRPO’s stationary policies and point out that GRPO’s way of aggregating preferences differs fundamentally from the logarithmic pooling of standard RLHF — affects not only the variance of the estimate but also the fixed point of the normalized objective itself. This is analytical work, and the conditions under which its conclusions hold (stationarity, infinitely many steps) are some distance from engineering practice, but it is a reminder of one thing: understanding purely as a “sampling budget” is incomplete.
Counterpoint: Are 2 Responses Enough?
The argument of 2510.00977 is worth restating on its own terms: GRPO’s effectiveness comes from an implicit contrastive objective — comparing just two responses already yields a learning signal, and the group mean is merely a control variate for variance reduction that does not need to be a precise statistical estimate, so 2-GRPO with also works and, structurally, even ends up connecting to DPO. Their theoretical analysis and experiments (at small to medium scale) both support this conclusion.
Does this contradict DeepSeekMath’s ? Not necessarily. Of the things a large buys, baseline precision is only one, and possibly the least important. The others are coverage of boundary-difficulty questions (the second role), the reliability of the filtering decision (the companion to dynamic sampling), and exploration coverage — a group needs at least one correct response for the model to have a “good sample” to learn from; for a hard question with , there is an 81% chance at that the group contains no positive sample, versus only 0.1% at . These gains depend strongly on model scale and on the task difficulty distribution: for a small model grinding through GSM8K-level problems, the distribution is concentrated in the middle range, so a small costs little; for a large model attacking AIME-level problems, is pinned around 0.1, every positive sample is precious, and is the exploration budget itself. Whether the 2-GRPO conclusion extends to large models in long-CoT settings is something nobody can answer yet (the paper’s experiments are limited in scale; this is a boundary, not a criticism).
The Cost Ledger, and My Take
First, the accounting: rollouts per training step = number of prompts × . DeepSeekMath’s configuration (batch 1024, ) gets through only 16 prompts per step — prompt diversity and response diversity are traded against each other on the same budget sheet. One more configuration detail deserves attention: DeepSeekMath performs only one policy update per exploration stage (strictly on-policy, §4.2), so the ratio is always 1 and clipping never triggers, and ’s entire effect shows up in the baseline and in coverage; whereas with DAPO/GSPO-style multi-mini-batch updates, also indirectly affects the clip fraction through std noise. The same does not have the same marginal return in the two training regimes.
My take: there is no universally optimal , because it spends money on three budget sheets at once — baseline precision (, with the fastest-diminishing returns), effective-group coverage (, with the most durable returns at boundary difficulty), and the exploration budget (the probability that a group contains a positive sample, almost linear on hard questions). The practical default (8–16) is the balance point under “a medium-difficulty distribution + dynamic sampling as a safety net”; 64 was the naive choice of the DeepSeekMath era, before dynamic sampling and before any clip modifications; 2 is the theoretical limit from the “contrastive objective” perspective, which in engineering terms nobody has yet dared to use at production scale. Before picking , first ask yourself which of the three sheets is the bottleneck: if the questions are too hard (the exploration budget is tight), increase or add dynamic sampling; if the questions are too varied (prompt diversity is tight), give the budget to the number of prompts; if training is unstable, check the std and the clip fraction first, and don’t rush to increase .
The first four posts in this series happen to form a single thread: Clip-Higher fixes the token-level clipping bounds, GSPO changes the unit of the ratio, TIS plugs the numerical gap between training and inference, and this post goes back to the source of the signal — how to allocate the sampling budget. Every “self-evident” default configuration in GRPO, once taken apart, turns out to have a ledger of its own.
Open Questions
- There is no public systematic ablation of the optimal coupling between , the learning rate, and the clip ; whether small + a large learning rate can match the signal-to-noise ratio of a large is backed only by scattered folk wisdom;
- No paper has quantified the actual impact on data efficiency of dynamic sampling’s “mistaken discard” rate () on medium-difficulty questions;
- Whether 2-GRPO holds for large models, long CoT, and boundary-difficulty tasks is one of the most worthwhile public replications to attempt right now.
References
- DeepSeekMath / original GRPO paper: arXiv:2402.03300; , batch 1024, and the single-step on-policy update are in §4.2
- DAPO paper: arXiv:2503.14476; the Dynamic Sampling constraint is in Equation 8, the ±1 reward in Equation 7
- It Takes Two: Your GRPO Is Secretly DPO: arXiv:2510.00977
- What is the Alignment Objective of GRPO?: arXiv:2502.18548
- RLHF Book, Chapter 6 (the baseline unbiasedness condition, the RLOO leave-one-out formula):
book/chapters/06-policy-gradients.md; teaching-code defaultnum_rollouts: 8:code/policy_gradients/configs/ - The zero-gradient-group probability table, the mistaken discards, and the clip-fraction chain are this post’s own calculations/inferences, each flagged where it appears in the text
- Previous post: The Ratio’s Wrong Unit: GSPO Goes Sequence-Level