<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>rlvr on Njx&#39;Log</title>
    <link>https://njx-njx.github.io/en/tags/rlvr/</link>
    <description>Recent content in rlvr on Njx&#39;Log</description>
    <image>
      <title>Njx&#39;Log</title>
      <url>https://njx-njx.github.io/blog-cover.jpg</url>
      <link>https://njx-njx.github.io/blog-cover.jpg</link>
    </image>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate>Tue, 27 Oct 2026 15:14:17 +0800</lastBuildDate>
    <atom:link href="https://njx-njx.github.io/en/tags/rlvr/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Judging Truncated Answers: DAPO&#39;s Overlong Fix</title>
      <link>https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/</link>
      <pubDate>Sun, 30 Aug 2026 12:05:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/</guid>
      <description>Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.</description>
      <content:encoded><![CDATA[<p>RL training needs a generation length cap, and truncated responses are treated as wrong by default—which misjudges &lsquo;undecidable&rsquo; samples as &lsquo;wrong&rsquo; ones and injects reward noise straight into the group-relative advantage. DAPO&rsquo;s two fixes: Overlong Filtering acknowledges that such samples are undecidable and masks their loss; Soft Overlong Punishment builds a linear buffer ramp before the cliff, turning length control into a continuous signal. This post unpacks the mechanism, the formula, and the configuration values, and fits it together with Dr.GRPO and token-mean into a complete picture of the length problem.</p>
<blockquote>
<p>This post comes with a companion <a href="https://njx-njx.github.io/reports/dapo-overlong-reward-shaping/">interactive report</a>
 (in Chinese), featuring a drill-down length ruler, a three-piece breakdown of Equation 13, and a switchable comparison of the three handling approaches.</p>
</blockquote>
<h2 id="an-engineering-setting-you-cant-avoid">An Engineering Setting You Can&rsquo;t Avoid</h2>
<p>Rollouts in RL training must have a generation length cap. GPU memory has to be reserved for the longest sequence, the batch has to wait for the slowest response to finish, and a single runaway response can drag down the throughput of the whole batch. DAPO&rsquo;s configuration: an expected maximum length of 16,384 tokens, an additional 4,096-token buffer, and a hard generation cap of 20,480 (paper §4.1).</p>
<p>Once a cap is set, some responses will inevitably be truncated. What should happen to these samples? The default practice is common: assign a punitive reward and treat them as wrong answers (reward recorded as 0 or a negative score).</p>
<p>The DAPO paper (arXiv:2503.14476) points out in §3.4 that this default causes problems: <strong>improper handling of truncated samples introduces reward noise, and that noise is injected into the entire training process, amplified by within-group normalization</strong>. The paper&rsquo;s reasoning is straightforward: when a reasoning process that is sound in itself gets penalized merely for running too long, the penalty confuses the model—it can&rsquo;t tell whether its reasoning was valid.</p>
<h2 id="how-the-noise-gets-into-the-advantage">How the Noise Gets into the Advantage</h2>
<p>GRPO&rsquo;s advantage is a within-group relative ranking: sample <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses for the same problem and subtract the group mean (dividing by the std). This means that <strong>once the reward of any single response is wrong, it affects the whole group</strong>.</p>
<p>Take a concrete example: 8 responses are sampled for a problem, and the true split is 4 right and 4 wrong; among them, 2 &ldquo;wrong&rdquo; responses are actually high-quality reasoning that got truncated (had they been finished, they would likely have been right).</p>
<ul>
<li>By true quality: the group mean is 0.5, the correct responses get positive advantages, and the wrong ones get negative ones;</li>
<li>Scoring truncation as 0: the truncated samples are also counted as wrong answers and the group mean is dragged down (6 counted as wrong, mean 0.25), so the advantage of every correct sample is inflated and the negative advantage of every wrong sample is softened—including the two that genuinely got it wrong.</li>
</ul>
<p>As if one layer of distortion weren&rsquo;t enough, there is a second: those two truncated high-quality reasoning traces themselves get reinforced as negative samples. What the model learns from them is not &ldquo;write a bit shorter&rdquo; but &ldquo;my way of reasoning is wrong&rdquo;—which is exactly where the paper says the model gets confused. Once these two kinds of penalty are lumped together, the model throws out the good way of reasoning along with &ldquo;writing long&rdquo;.</p>
<h2 id="approach-1-overlong-filteringdeem-truncated-samples-undecidable-and-mask-their-loss">Approach 1: Overlong Filtering—Deem Truncated Samples &ldquo;Undecidable&rdquo; and Mask Their Loss</h2>
<p>DAPO&rsquo;s first approach: rather than treating a truncated sample as a wrong sample, simply refuse to recognize it as a sample at all. Overlong Filtering masks the entire loss of a truncated response—the sample takes no part in advantage estimation and produces no gradient.</p>
<p>The effect shown in Figure 5 of the paper: after applying Overlong Reward Shaping, both AIME accuracy and the actor model&rsquo;s entropy become more stable, and training stability improves markedly. This is worth noting: the stabilized entropy means truncation noise itself is interfering with the exploration dynamics—it doesn&rsquo;t merely affect the reward numbers on paper.</p>
<p>The costs are just as obvious. The paper doesn&rsquo;t go into them, so I&rsquo;ll add two points:</p>
<ol>
<li><strong>Wasted compute</strong>. The rollout cost of a truncated sample has already been paid (it is often the longest response, which happens to consume the most compute), and once it is masked, not a single gradient comes back in return. This is the same logic as dynamic sampling—both &ldquo;trade compute for signal quality&rdquo;, which is the orientation of the whole DAPO paper;</li>
<li><strong>Hidden survivorship bias</strong>. Which samples are more likely to be truncated? Long problems, hard problems, problems that need long CoT. Systematically masking truncated samples amounts to systematically down-weighting hard problems in the training distribution. In the short term, training becomes more stable; the long-term cost is that the model may reveal weaknesses on exactly the problems that most need long reasoning. The paper doesn&rsquo;t discuss this side effect; this reservation is my own.</li>
</ol>
<h2 id="approach-2-soft-overlong-punishmentbuild-a-buffer-ramp-before-the-cliff">Approach 2: Soft Overlong Punishment—Build a Buffer Ramp Before the Cliff</h2>
<p>Masking everything loses information: the model can&rsquo;t learn that length itself matters. DAPO&rsquo;s second approach is Soft Overlong Punishment, which turns length itself into a <strong>continuous, discriminable reward dimension</strong> (Equation 13 in the paper):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>R</mi><mrow><mi>l</mi><mi>e</mi><mi>n</mi><mi>g</mi><mi>t</mi><mi>h</mi></mrow></msub><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mo fence="true">{</mo><mtable rowspacing="0.36em" columnalign="left left" columnspacing="1em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mn>0</mn><mo separator="true">,</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi><mo>≤</mo><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>−</mo><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mstyle displaystyle="true" scriptlevel="0"><mfrac><mrow><mo stretchy="false">(</mo><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>−</mo><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub><mo stretchy="false">)</mo><mo>−</mo><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi></mrow><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub></mfrac></mstyle><mo separator="true">,</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>−</mo><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub><mo>&lt;</mo><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi><mo>≤</mo><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mo>−</mo><mn>1</mn><mo separator="true">,</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>&lt;</mo><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi></mrow></mstyle></mtd></mtr></mtable></mrow></mrow><annotation encoding="application/x-tex">R_{length}(y) = \begin{cases} 0, &amp; |y| \le L_{max} - L_{cache} \\ \dfrac{(L_{max} - L_{cache}) - |y|}{L_{cache}}, &amp; L_{max}-L_{cache} &lt; |y| \le L_{max} \\ -1, &amp; L_{max} &lt; |y| \end{cases}</annotation></semantics></math></span><p>The three pieces, one at a time:</p>
<ul>
<li><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi><mo>≤</mo><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>−</mo><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub></mrow><annotation encoding="application/x-tex">|y| \le L_{max}-L_{cache}</annotation></semantics></math></span> (i.e., ≤ 12,288 tokens under DAPO&rsquo;s configuration): no penalty. Length is completely free in this regime;</li>
<li>Entering the cache interval (12,288 to 16,384): a linearly ramping penalty, falling smoothly from 0 to -1. Every extra token written increases the penalty by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub><mo>=</mo><mn>1</mn><mi mathvariant="normal">/</mi><mn>4096</mn></mrow><annotation encoding="application/x-tex">1/L_{cache} = 1/4096</annotation></semantics></math></span>;</li>
<li>Beyond <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub></mrow><annotation encoding="application/x-tex">L_{max}</annotation></semantics></math></span> (truncated): the penalty is capped at -1. Note that this term is <strong>added to the original correctness reward</strong>—a response that is correct but runs over can still have its total score pulled down.</li>
</ul>
<p>The most ingenious thing about this design: <strong>the cliff is not removed; a buffer ramp is built in front of it</strong>. Going past <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub></mrow><annotation encoding="application/x-tex">L_{max}</annotation></semantics></math></span> is still a hard -1 penalty, just as harsh as the naive approach; but the model starts receiving a continuous &ldquo;almost over budget&rdquo; signal four thousand tokens before the cliff, and the signal strength is monotonically related to how far over budget it is. After within-group normalization, two responses—&ldquo;12,500 tokens, nearly over&rdquo; and &ldquo;16,300 tokens, about to go over&rdquo;—get a distinguishable advantage gap, from which the model can learn to <strong>wrap up early</strong> instead of only feeling the pain once it hits the cap. Length control turns from a binary event (not over / over) into a continuous, learnable task.</p>
<h2 id="placing-it-in-the-full-length-problem-picture">Placing It in the Full &ldquo;Length Problem&rdquo; Picture</h2>
<p>The previous few posts in this series all had to do with length; this post covers the final aspect:</p>
<ul>
<li><strong>Gradient dilution from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y|</annotation></semantics></math></span></strong> (Dr.GRPO, see the <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">normalization-bias post</a>
): the longer a wrong response, the smaller the penalty on each token, so the model tends to make wrong responses overly long. Fix: change the normalization term;</li>
<li><strong>token-mean</strong> (DAPO §3.3): instead of averaging within each sample first and then summing over the group, average uniformly over all tokens in the batch, which keeps long responses from taking an outsized weight in the loss. Fix: change the loss aggregation;</li>
<li><strong>Overlong</strong> (DAPO §3.4, this post): the truncated samples produced by the generation length cap have undecidable rewards. Fix: mask them (filtering) or change the reward function (soft punishment).</li>
</ul>
<p>The three address length problems at three different levels: gradient weighting, loss aggregation, and reward judgment. DAPO uses both token-mean and overlong shaping, meaning it fixes both the gradient side and the reward side; Dr.GRPO focuses on normalization. They don&rsquo;t conflict.</p>
<p>One can also see a design orientation in DAPO: <strong>better to use no signal than an unreliable one</strong>. Dynamic sampling filters out prompts with no gradient signal (all-correct / all-wrong problems), overlong filtering masks samples with unreliable rewards (truncated responses), and Dataset Transformation converts all answers to integers to eliminate parser noise (§3.5, DAPO-Math-17K). All four techniques point in the same direction: training collapse is often not because the algorithm isn&rsquo;t good enough, but because noise has crept into the signal.</p>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>truncated samples are &ldquo;undecidable&rdquo; samples, and judging them as wrong is a misjudgment at the reward level</strong>. The premise of RLVR is that rewards are verifiable—you assign a score only if you can judge right or wrong. For a truncated sample, the verifier never got to finish its judgment at all; it is neither a 1 nor a 0. Overlong Filtering acknowledges this undecidability, while Soft Punishment introduces a new decidable dimension (length itself). Both approaches are consistent with the principle that &ldquo;rewards must be verifiable&rdquo;, whereas the naive punitive reward violates it.</p>
<p>Second, <strong>the fact that entropy is more stable after overlong handling matters more than it appears</strong>. Figure 5(b) shows that the actor&rsquo;s entropy curve is smoother after shaping. This indicates that the chaos near the length cap (high-quality long reasoning that suddenly gets penalized, responses forced into hasty endings) doesn&rsquo;t merely distort the reward; it also adds exploration noise. Read this together with the <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Clip-Higher post</a>
 (entropy collapse): entropy can collapse when wrongly suppressed by the optimizer, and it can also swing wildly when disturbed by reward noise. Entropy monitoring has to be analyzed together with the quality of reward judgments.</p>
<p>Third, <strong>the cache width and the masking ratio are two hyperparameters that need monitoring, and there is currently no public basis for them</strong>. The paper doesn&rsquo;t say how the buffer-ramp length of 4,096/16,384 = 25% was chosen; during training, the truncation rate (the fraction of masked samples) drifts with the model&rsquo;s length behavior—when the masking rate is too high, the effective batch shrinks and the share of hard problems drops; when it is too low, that indicates length control has already broken down or the cap is set too loose. The DAPO paper has no monitoring curves for either number, so reproductions have to fill them in themselves.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>There are no public experiments on whether the hard-problem weight lost to Overlong Filtering needs to be compensated with a progressive length curriculum in long training runs;</li>
<li>The rationale and ablations for the choice of cache width (25%) are not disclosed in the paper;</li>
<li>There is currently no data on whether long but correct exploratory responses get doubly suppressed when Soft Punishment&rsquo;s -1 cap is stacked with Clip-Higher&rsquo;s asymmetric clipping.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>DAPO paper: <a href="https://arxiv.org/abs/2503.14476">arXiv:2503.14476v2</a>
 (source of the §3.4 mechanism description, Equation 13, the §4.1 configuration values, and Figure 5)</li>
<li>Textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
: Chapter 18 <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter18_grpo/deepseek-dapo.md"><code>docs/chapter18_grpo/deepseek-dapo.md</code></a>
 and the appendix DAPO cheatsheet (<a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/en/appendix_code_cheatsheet/dapo.md"><code>docs/en/appendix_code_cheatsheet/dapo.md</code></a>
)</li>
<li>Previous posts: <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
, <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">&ldquo;When Wrong, Write Longer&rdquo;: How GRPO&rsquo;s Two Normalization Terms Distort the Optimization Direction</a>
</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>Samples per Prompt: What GRPO&#39;s Group Size Buys</title>
      <link>https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/</link>
      <pubDate>Sat, 29 Aug 2026 12:00:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/</guid>
      <description>64, 8 or 2 samples per prompt? G is GRPO&amp;rsquo;s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.</description>
      <content:encoded><![CDATA[<p>DeepSeekMath samples 64 responses per question, a teaching implementation defaults to 8, and the latest paper claims even 2 is enough — a 32x spread on the same knob. G is GRPO&rsquo;s only source of a baseline and its biggest compute lever, so what does it actually buy? This post unpacks G&rsquo;s three roles: the sample size of the control variate, the coverage of groups with effective gradients, and the quality of the std-normalization denominator, plus why dynamic sampling can replace a larger G with filtering.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/grpo-group-size-tradeoffs/">interactive report</a>
 (in Chinese), with a zero-gradient-group probability calculator, a dual-curve baseline chart, and a side-by-side comparison of the three camps&rsquo; choices.</p>
</blockquote>
<h2 id="a-knob-with-a-32x-spread">A Knob With a 32x Spread</h2>
<p>On every training step, GRPO samples a group of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses for each prompt. How large this <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> should be is a question on which different camps give wildly different answers:</p>
<ul>
<li>DeepSeekMath (the original GRPO paper, arXiv:2402.03300 §4.2): <strong>64</strong>. With a batch size of 1024, that means a single step covers only 16 prompts;</li>
<li>The RLHF Book&rsquo;s companion teaching code (<code>code/policy_gradients/configs/</code>): <strong>8</strong>, the default across the whole lineup, from GRPO to GSPO and CISPO;</li>
<li>The October 2025 paper &ldquo;It Takes Two: Your GRPO Is Secretly DPO&rdquo; (arXiv:2510.00977): <strong>2</strong> is enough, and the paper argues specifically that the &ldquo;large groups are necessary&rdquo; view is a misreading of why GRPO succeeds.</li>
</ul>
<p>A 32x gap. Worse, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is two things at once: it is GRPO&rsquo;s only source of a baseline (there is no critic; the group mean is the baseline), and it is the biggest multiplier on training cost (rollouts are the main compute expense in RL training, and doubling <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> doubles that cost). One knob governs both signal quality and the compute bill, yet people have gotten everything from 2 to 64 to work — which suggests that the question &ldquo;what does <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> actually buy?&rdquo; is itself worth taking apart.</p>
<h2 id="role-one-the-baselines-sample-size">Role One: The Baseline&rsquo;s Sample Size</h2>
<p>Having dropped the critic, GRPO uses the group mean as its baseline: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>=</mo><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mtext>mean</mtext><mo stretchy="false">(</mo><mo stretchy="false">{</mo><msub><mi>r</mi><mi>j</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\hat{A}_i = r_i - \text{mean}(\{r_j\}_{j=1}^G)</annotation></semantics></math></span> (setting aside the division by std for now). Subtracting a baseline is the standard variance-reduction technique for policy gradients, and the validity condition the textbook (RLHF Book, Chapter 6) gives is that the baseline must not depend on the current action, because <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="double-struck">E</mi><mrow><mi>a</mi><mo>∼</mo><mi>π</mi></mrow></msub><mo stretchy="false">[</mo><mi>b</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>a</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\mathbb{E}_{a\sim\pi}[b(s)\nabla_\theta\log\pi_\theta(a|s)] = 0</annotation></semantics></math></span> holds for any <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">b(s)</annotation></semantics></math></span> that depends only on the state — the baseline leaves the expected gradient unchanged and only suppresses variance.</p>
<p>There is a detail here that few people mention: <strong>GRPO&rsquo;s mean includes the very response being evaluated</strong>, so strictly speaking the baseline depends on the action, and the unbiasedness condition is not strictly satisfied. Compare RLOO (REINFORCE Leave-One-Out, same chapter of rlhf-book): its baseline is the mean of the <strong>other</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">K-1</annotation></semantics></math></span> responses, deliberately excluding the response itself, to &ldquo;keep the gradient estimator fully unbiased.&rdquo; The relationship between the two advantages can be written out directly:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msubsup><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi><mtext>RLOO</mtext></msubsup><mo>=</mo><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mfrac><mn>1</mn><mrow><mi>G</mi><mo>−</mo><mn>1</mn></mrow></mfrac><munder><mo>∑</mo><mrow><mi>j</mi><mo mathvariant="normal">≠</mo><mi>i</mi></mrow></munder><msub><mi>r</mi><mi>j</mi></msub><mo>=</mo><mfrac><mi>G</mi><mrow><mi>G</mi><mo>−</mo><mn>1</mn></mrow></mfrac><mrow><mo fence="true">(</mo><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munder><mo>∑</mo><mi>j</mi></munder><msub><mi>r</mi><mi>j</mi></msub><mo fence="true">)</mo></mrow><mo>=</mo><mfrac><mi>G</mi><mrow><mi>G</mi><mo>−</mo><mn>1</mn></mrow></mfrac><msubsup><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi><mtext>GRPO(no std)</mtext></msubsup></mrow><annotation encoding="application/x-tex">\hat{A}_i^{\text{RLOO}} = r_i - \frac{1}{G-1}\sum_{j\neq i}r_j = \frac{G}{G-1}\left(r_i - \frac{1}{G}\sum_{j}r_j\right) = \frac{G}{G-1}\hat{A}_i^{\text{GRPO(no std)}}</annotation></semantics></math></span><p>The directions are identical; the two differ only by a scale factor of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mi>G</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">G/(G-1)</annotation></semantics></math></span>. At <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span> the factor is 1.016, which hardly matters; at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">G=8</annotation></semantics></math></span> it is 1.14; <strong>at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">G=2</annotation></semantics></math></span> it is 2</strong> — the GRPO-form advantage is only half as large as the leave-one-out form. This bias is not directional, just a rescaling that grows as <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> shrinks, and in practice its effect can be absorbed by the learning rate (this is an inference, not an experimental finding from any paper), but it explains why every property of small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> gets amplified.</p>
<p>The second entry in the ledger is the baseline&rsquo;s estimation noise. The group mean is an estimate of the true expectation <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>r</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\mathbb{E}[r]</annotation></semantics></math></span> from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> samples, with noise of magnitude roughly <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mi>r</mi></msub><mi mathvariant="normal">/</mi><msqrt><mi>G</mi></msqrt></mrow><annotation encoding="application/x-tex">\sigma_r/\sqrt{G}</annotation></semantics></math></span> (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mi>r</mi></msub></mrow><annotation encoding="application/x-tex">\sigma_r</annotation></semantics></math></span> is the standard deviation of a single response&rsquo;s reward). When <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is small, this noise leaks straight into every response&rsquo;s advantage: a correct response can end up with a negative advantage because &ldquo;everyone else in its group happened to be strong too.&rdquo; The probability that baseline noise randomly flips the gradient direction rises as <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> decreases — this is the standard &ldquo;large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> reduces variance&rdquo; argument, and also what 2510.00977 sets out to challenge; more on that later.</p>
<h2 id="role-two-coverage-of-groups-with-effective-gradients">Role Two: Coverage of Groups With Effective Gradients</h2>
<p>Rewards in RLVR are binary (right or wrong; DAPO uses +1/−1, DeepSeekMath uses 1/0, and the two are equivalent). If a group&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses are all right or all wrong, there is no reward variance within the group and every advantage is zero — those <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> rollouts were run for nothing and contribute no gradient at all.</p>
<p>The probability that a group is uniform can be computed directly (what follows is my own calculation): let the model&rsquo;s per-sample accuracy on the question be <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span>, with samples independent; then</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><mtext>zero-gradient group</mtext><mo stretchy="false">)</mo><mo>=</mo><msup><mi>p</mi><mi>G</mi></msup><mo>+</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><msup><mo stretchy="false">)</mo><mi>G</mi></msup></mrow><annotation encoding="application/x-tex">P(\text{zero-gradient group}) = p^G + (1-p)^G</annotation></semantics></math></span><p>Plugging in numbers to see the trend (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span> is the per-sample accuracy; the table gives the probability that the group has zero gradient):</p>
<table>
  <thead>
      <tr>
          <th><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span></th>
          <th><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">G=8</annotation></semantics></math></span></th>
          <th><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span></th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>0.5</td>
          <td>0.8%</td>
          <td>≈0</td>
      </tr>
      <tr>
          <td>0.8</td>
          <td>16.8%</td>
          <td>≈0</td>
      </tr>
      <tr>
          <td>0.9</td>
          <td>43.0%</td>
          <td>0.1%</td>
      </tr>
      <tr>
          <td>0.95</td>
          <td>66.3%</td>
          <td>3.8%</td>
      </tr>
      <tr>
          <td>0.99</td>
          <td>92.3%</td>
          <td>52.6%</td>
      </tr>
  </tbody>
</table>
<p>How to read it: for medium-difficulty questions (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span> in the 0.3–0.7 range), even a small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is basically safe; what really consumes <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is <strong>questions at the difficulty boundary</strong> — at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>=</mo><mn>0.95</mn></mrow><annotation encoding="application/x-tex">p=0.95</annotation></semantics></math></span>, with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">G=8</annotation></semantics></math></span> two-thirds of the groups are all-correct and gradient-free, and only <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span> pushes that down to 4%. And the training trajectory pushes questions precisely toward the boundaries: for questions the model has learned, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>→</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">p \to 1</annotation></semantics></math></span>; for questions it can&rsquo;t learn, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>→</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">p \to 0</annotation></semantics></math></span>; and the later in training, the fewer questions remain in the &ldquo;gradient-bearing range.&rdquo; This is one source of the &ldquo;training efficiency and stability&rdquo; problems that the DAPO paper describes in its motivation.</p>
<p>Faced with the same table, there are two remedies. One is to increase <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> and brute-force <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>p</mi><mi>G</mi></msup></mrow><annotation encoding="application/x-tex">p^G</annotation></semantics></math></span> down — with costs growing multiplicatively. The other is DAPO&rsquo;s <strong>Dynamic Sampling</strong>: instead of increasing <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>, simply rule that zero-gradient groups are not accepted. DAPO&rsquo;s objective includes a hard constraint (the s.t. condition of Equation 8 in the paper):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>0</mn><mo>&lt;</mo><mrow><mo fence="true">∣</mo><mo stretchy="false">{</mo><msub><mi>o</mi><mi>i</mi></msub><mo>∣</mo><mtext mathvariant="monospace">is_equivalent</mtext><mo stretchy="false">(</mo><mi>a</mi><mo separator="true">,</mo><msub><mi>o</mi><mi>i</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">}</mo><mo fence="true">∣</mo></mrow><mo>&lt;</mo><mi>G</mi></mrow><annotation encoding="application/x-tex">0 &lt; \left|\{o_i \mid \texttt{is\_equivalent}(a, o_i)\}\right| &lt; G</annotation></semantics></math></span><p>That is, the number of correct responses in a group must lie strictly between 0 and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> — all-correct and all-wrong groups are filtered out and replaced through oversampling, guaranteeing that every prompt in the batch has an effective gradient. For the same effective-group coverage, the price of filtering is additive oversampling rather than a multiplicative increase in <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>.</p>
<p>But there is another layer of coupling between filtering and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> that is easy to miss: <strong>the filtering decision is itself based on an estimate from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> samples</strong>. At <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">G=4</annotation></semantics></math></span>, &ldquo;all 4 wrong&rdquo; does not mean the model can&rsquo;t solve the question — a question with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>=</mo><mn>0.3</mn></mrow><annotation encoding="application/x-tex">p=0.3</annotation></semantics></math></span> has a 24% chance of getting all 4 wrong and being mistakenly thrown on the scrap heap. Pairing a small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> with dynamic sampling will misjudge a whole set of medium-difficulty questions that &ldquo;actually carry gradient value&rdquo; as dead groups; the misjudgment rate falls exponentially with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><msup><mo stretchy="false">)</mo><mi>G</mi></msup></mrow><annotation encoding="application/x-tex"> (1-p)^G </annotation></semantics></math></span>, the same table). So dynamic sampling does not eliminate <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>&rsquo;s coverage problem; it merely converts it from &ldquo;zero gradient&rdquo; into &ldquo;mistaken discards&rdquo; (this point is an inference; the DAPO paper does not discuss the misjudgment rate).</p>
<h2 id="role-three-the-quality-of-the-std-normalization-denominator">Role Three: The Quality of the std-Normalization Denominator</h2>
<p>GRPO&rsquo;s advantage is also divided by the group std: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>=</mo><mo stretchy="false">(</mo><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mtext>mean</mtext><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mtext>std</mtext></mrow><annotation encoding="application/x-tex">\hat{A}_i = (r_i - \text{mean})/\text{std}</annotation></semantics></math></span>. This std is likewise an estimate from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> samples and is very noisy when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is small; for a uniform group the std is exactly 0, so implementations either add an eps or skip the group (setting a divide-by-zero trap).</p>
<p>Dr.GRPO&rsquo;s critique of the std (see the <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">dedicated post</a>
) — that when the std of all-correct or all-wrong groups approaches zero, the advantage is inflated to unreasonable magnitudes — is at its worst with small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>: when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is small, the std is more likely to land on a small value, and extreme advantages show up frequently. Extreme advantages in turn push up how far the ratio moves, indirectly raising the clip fraction (this chain is an inference: abnormally large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span> → <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_\theta</annotation></semantics></math></span> moves more within a single step → <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">w_t</annotation></semantics></math></span> hits the clipping bound sooner). In other words, part of the claim that &ldquo;large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> stabilizes training&rdquo; is really not about the baseline but about the std — two ledgers that the literature often lumps together.</p>
<p>One more theoretical result worth knowing (arXiv:2502.18548, &ldquo;What is the Alignment Objective of GRPO?&rdquo;): the authors give a framework characterizing GRPO&rsquo;s stationary policies and point out that GRPO&rsquo;s way of aggregating preferences differs fundamentally from the logarithmic pooling of standard RLHF — <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> affects not only the variance of the estimate but also the fixed point of the normalized objective itself. This is analytical work, and the conditions under which its conclusions hold (stationarity, infinitely many steps) are some distance from engineering practice, but it is a reminder of one thing: understanding <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> purely as a &ldquo;sampling budget&rdquo; is incomplete.</p>
<h2 id="counterpoint-are-2-responses-enough">Counterpoint: Are 2 Responses Enough?</h2>
<p>The argument of 2510.00977 is worth restating on its own terms: GRPO&rsquo;s effectiveness comes from an <strong>implicit contrastive objective</strong> — comparing just two responses already yields a learning signal, and the group mean is merely a control variate for variance reduction that does not need to be a precise statistical estimate, so 2-GRPO with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">G=2</annotation></semantics></math></span> also works and, structurally, even ends up connecting to DPO. Their theoretical analysis and experiments (at small to medium scale) both support this conclusion.</p>
<p>Does this contradict DeepSeekMath&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span>? Not necessarily. Of the things a large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> buys, baseline precision is only one, and possibly the least important. The others are coverage of boundary-difficulty questions (the second role), the reliability of the filtering decision (the companion to dynamic sampling), and exploration coverage — a group needs at least one correct response for the model to have a &ldquo;good sample&rdquo; to learn from; for a hard question with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>=</mo><mn>0.1</mn></mrow><annotation encoding="application/x-tex">p=0.1</annotation></semantics></math></span>, there is an 81% chance at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">G=2</annotation></semantics></math></span> that the group contains no positive sample, versus only 0.1% at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span>. These gains depend strongly on model scale and on the task difficulty distribution: for a small model grinding through GSM8K-level problems, the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span> distribution is concentrated in the middle range, so a small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> costs little; for a large model attacking AIME-level problems, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span> is pinned around 0.1, every positive sample is precious, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is the exploration budget itself. Whether the 2-GRPO conclusion extends to large models in long-CoT settings is something nobody can answer yet (the paper&rsquo;s experiments are limited in scale; this is a boundary, not a criticism).</p>
<h2 id="the-cost-ledger-and-my-take">The Cost Ledger, and My Take</h2>
<p>First, the accounting: rollouts per training step = number of prompts × <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>. DeepSeekMath&rsquo;s configuration (batch 1024, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span>) gets through only 16 prompts per step — prompt diversity and response diversity are traded against each other on the same budget sheet. One more configuration detail deserves attention: DeepSeekMath performs only <strong>one</strong> policy update per exploration stage (strictly on-policy, §4.2), so the ratio is always 1 and clipping never triggers, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>&rsquo;s entire effect shows up in the baseline and in coverage; whereas with DAPO/GSPO-style multi-mini-batch updates, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> also indirectly affects the clip fraction through std noise. The same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> does not have the same marginal return in the two training regimes.</p>
<p>My take: <strong>there is no universally optimal <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>, because it spends money on three budget sheets at once</strong> — baseline precision (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mi>r</mi></msub><mi mathvariant="normal">/</mi><msqrt><mi>G</mi></msqrt></mrow><annotation encoding="application/x-tex">\sigma_r/\sqrt{G}</annotation></semantics></math></span>, with the fastest-diminishing returns), effective-group coverage (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>p</mi><mi>G</mi></msup></mrow><annotation encoding="application/x-tex">p^G</annotation></semantics></math></span>, with the most durable returns at boundary difficulty), and the exploration budget (the probability that a group contains a positive sample, almost linear on hard questions). The practical default (8–16) is the balance point under &ldquo;a medium-difficulty distribution + dynamic sampling as a safety net&rdquo;; 64 was the naive choice of the DeepSeekMath era, before dynamic sampling and before any clip modifications; 2 is the theoretical limit from the &ldquo;contrastive objective&rdquo; perspective, which in engineering terms nobody has yet dared to use at production scale. Before picking <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>, first ask yourself which of the three sheets is the bottleneck: if the questions are too hard (the exploration budget is tight), increase <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> or add dynamic sampling; if the questions are too varied (prompt diversity is tight), give the budget to the number of prompts; if training is unstable, check the std and the clip fraction first, and don&rsquo;t rush to increase <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>.</p>
<p>The first four posts in this series happen to form a single thread: Clip-Higher fixes the token-level clipping bounds, GSPO changes the unit of the ratio, TIS plugs the numerical gap between training and inference, and this post goes back to the source of the signal — how to allocate the sampling budget. Every &ldquo;self-evident&rdquo; default configuration in GRPO, once taken apart, turns out to have a ledger of its own.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>There is no public systematic ablation of the optimal coupling between <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>, the learning rate, and the clip <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span>; whether small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> + a large learning rate can match the signal-to-noise ratio of a large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is backed only by scattered folk wisdom;</li>
<li>No paper has quantified the actual impact on data efficiency of dynamic sampling&rsquo;s &ldquo;mistaken discard&rdquo; rate (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><msup><mo stretchy="false">)</mo><mi>G</mi></msup></mrow><annotation encoding="application/x-tex">(1-p)^G</annotation></semantics></math></span>) on medium-difficulty questions;</li>
<li>Whether 2-GRPO holds for large models, long CoT, and boundary-difficulty tasks is one of the most worthwhile public replications to attempt right now.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>DeepSeekMath / original GRPO paper: <a href="https://arxiv.org/abs/2402.03300">arXiv:2402.03300</a>
; <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span>, batch 1024, and the single-step on-policy update are in §4.2</li>
<li>DAPO paper: <a href="https://arxiv.org/abs/2503.14476">arXiv:2503.14476</a>
; the Dynamic Sampling constraint is in Equation 8, the ±1 reward in Equation 7</li>
<li>It Takes Two: Your GRPO Is Secretly DPO: <a href="https://arxiv.org/abs/2510.00977">arXiv:2510.00977</a>
</li>
<li>What is the Alignment Objective of GRPO?: <a href="https://arxiv.org/abs/2502.18548">arXiv:2502.18548</a>
</li>
<li>RLHF Book, Chapter 6 (the baseline unbiasedness condition, the RLOO leave-one-out formula): <a href="https://github.com/natolambert/rlhf-book/blob/main/book/chapters/06-policy-gradients.md"><code>book/chapters/06-policy-gradients.md</code></a>
; teaching-code default <code>num_rollouts: 8</code>: <a href="https://github.com/natolambert/rlhf-book/tree/main/code/policy_gradients/configs"><code>code/policy_gradients/configs/</code></a>
</li>
<li>The zero-gradient-group probability table, the mistaken discards, and the clip-fraction chain are this post&rsquo;s own calculations/inferences, each flagged where it appears in the text</li>
<li>Previous post: <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">The Ratio&rsquo;s Wrong Unit: GSPO Goes Sequence-Level</a>
</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>Entropy Collapse: What DAPO&#39;s Clip-Higher Fixes</title>
      <link>https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/</link>
      <pubDate>Wed, 26 Aug 2026 14:30:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/</guid>
      <description>Naive GRPO collapses entropy fast. DAPO&amp;rsquo;s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.</description>
      <content:encoded><![CDATA[<p>When naive GRPO is used to reproduce R1, entropy collapses rapidly and samples within a group converge; DAPO&rsquo;s Clip-Higher merely raises the upper clipping bound from 0.2 to 0.28. Behind that one number lies a structural asymmetry of PPO-Clip on the probability axis. This post unpacks the failure chain, the mechanism, the evidence, and the limits.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/dapo-clip-higher-entropy-collapse/">interactive report</a>
 (in Chinese), with drill-down breakdowns of the formulas and charts of the ablation data.</p>
</blockquote>
<h2 id="a-scene-from-a-training-run">A Scene from a Training Run</h2>
<p>In early 2025, many teams were doing the same thing: reproducing DeepSeek-R1&rsquo;s training with GRPO. The DAPO paper (arXiv:2503.14476) reports the authors&rsquo; own starting point—running naive GRPO on the Qwen2.5-32B base model scored only 30 on AIME 2024, while DeepSeek&rsquo;s R1-Zero-Qwen-32B scored 47.</p>
<p>The gap did not open up gradually; it was a very specific way of dying: during training, the actor model&rsquo;s entropy drops rapidly, the responses sampled as a group for the same problem grow more and more alike, and in the end they are nearly identical. The paper calls this <strong>entropy collapse</strong>. Sampling loses its diversity, rewards within the group converge, GRPO&rsquo;s advantages all approach zero, and the gradient signal extinguishes itself. Training is still running, but it can no longer learn anything.</p>
<p>The first remedy DAPO prescribes is called <strong>Clip-Higher</strong>: raise PPO-Clip&rsquo;s upper clipping bound ε from the default 0.2 to 0.28 and leave the lower bound at 0.2. The change is a single number. Why can this one number stop the collapse? The answer lies in a structural property of PPO-Clip that is rarely spelled out.</p>
<h2 id="first-pinning-down-entropy">First, Pinning Down &ldquo;Entropy&rdquo;</h2>
<p>This whole post revolves around a single word—&ldquo;entropy&rdquo;—so let me pin down its definition first. The entropy here is exactly the entropy of information theory, and the object it applies to is <strong>the probability distribution over the entire vocabulary that the model outputs each time it generates a token</strong>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>H</mi><mo>=</mo><mo>−</mo><munder><mo>∑</mo><mrow><mi>x</mi><mo>∈</mo><mtext>vocab</mtext></mrow></munder><mi>p</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi>log</mi><mo>⁡</mo><mi>p</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">H = -\sum_{x \in \text{vocab}} p(x) \log p(x)</annotation></semantics></math></span><p>The flatter the distribution (many tokens with similar probabilities), the higher the entropy, and the more the model &ldquo;can&rsquo;t make up its mind&rdquo;; the sharper the distribution (one token takes 0.99 and the rest split 0.01), the lower the entropy, and the more &ldquo;certain&rdquo; the model is. A numerical example: suppose the vocabulary has only 3 tokens; the distribution [0.9, 0.05, 0.05] has an entropy of about 0.39 nat, while the uniform distribution [1/3, 1/3, 1/3] has about 1.10 nat. Sample 10 times from the former and you get the same token 9 times; samples from the latter spread out evenly—<strong>entropy is the mathematical measure of &ldquo;sampling diversity&rdquo;</strong>. The &ldquo;actor model entropy&rdquo; that the DAPO paper monitors is the average of these per-step distribution entropies over rollout generation.</p>
<p>Why it is the lifeline of RL training: all of GRPO&rsquo;s training signal comes from the within-group reward variance, and that variance comes from sampling diversity. With high entropy, 16 responses sampled for the same problem all differ, some right and some wrong, and the advantages are nonzero; when entropy collapses, the 16 converge, the rewards are all identical, and the advantages are all zero.</p>
<p>Two caveats, so as not to over-read this: first, <strong>higher entropy is not always better</strong>—excessively high entropy corresponds to runaway modes such as gibberish and repeated words; the paper only says entropy should be kept within an &ldquo;appropriate range&rdquo;, without saying how to determine that range (one of the open questions at the end of this post). Second, <strong>don&rsquo;t conflate entropy with sampling temperature</strong>: entropy is the intrinsic shape of the distribution, while temperature is how you draw from it; it takes parameters like temperature 1.0 at rollout time to cash entropy out as diversity. Clip-Higher affects the former—it keeps training itself from squeezing the distribution ever sharper.</p>
<h2 id="first-working-out-the-ppo-clip-math">First, Working Out the PPO-Clip Math</h2>
<p>The PPO/GRPO objective, viewed token by token, is:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mi>min</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><msub><mi>r</mi><mi>t</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mtext> </mtext><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub><mo separator="true">,</mo><mtext> clip</mtext><mo stretchy="false">(</mo><msub><mi>r</mi><mi>t</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo separator="true">,</mo><mtext> </mtext><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mtext> </mtext><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">)</mo><mtext> </mtext><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub><mo fence="true">)</mo></mrow><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">J(\theta) = \mathbb{E}\left[\min\left(r_t(\theta)\,\hat{A}_t,\ \text{clip}(r_t(\theta),\ 1-\varepsilon,\ 1+\varepsilon)\,\hat{A}_t\right)\right]</annotation></semantics></math></span><p>The symbols, one by one:</p>
<ul>
<li><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>r</mi><mi>t</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>o</mi><mi>t</mi></msub><mo>∣</mo><mi>q</mi><mo separator="true">,</mo><msub><mi>o</mi><mrow><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><msub><mi>o</mi><mi>t</mi></msub><mo>∣</mo><mi>q</mi><mo separator="true">,</mo><msub><mi>o</mi><mrow><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">r_t(\theta) = \pi_\theta(o_t \mid q, o_{&lt;t}) / \pi_{\theta_{old}}(o_t \mid q, o_{&lt;t})</annotation></semantics></math></span>, the <strong>importance sampling ratio</strong>: the current policy&rsquo;s probability for the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span>-th token divided by the old policy&rsquo;s probability at sampling time. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">r=1</annotation></semantics></math></span> means the policy has not changed.</li>
<li><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\hat{A}_t</annotation></semantics></math></span>, the <strong>advantage</strong>: how much better this token is than average. In GRPO it is computed by within-group normalization: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo>=</mo><mo stretchy="false">(</mo><msub><mi>R</mi><mi>i</mi></msub><mo>−</mo><mtext>mean</mtext><mo stretchy="false">(</mo><mo stretchy="false">{</mo><mi>R</mi><mo stretchy="false">}</mo><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mtext>std</mtext><mo stretchy="false">(</mo><mo stretchy="false">{</mo><mi>R</mi><mo stretchy="false">}</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\hat{A}_{i,t} = (R_i - \text{mean}(\{R\})) / \text{std}(\{R\})</annotation></semantics></math></span>.</li>
<li><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span>, the <strong>clipping range</strong>: how far the ratio is allowed to deviate from 1; the default in the vast majority of implementations is 0.2—a default that comes from the Atari and robotic-control experiments in the original 2017 PPO paper.</li>
</ul>
<p>On where the advantage comes from, PPO and GRPO take two different paths; don&rsquo;t mix them up when reading the literature:</p>
<ul>
<li><strong>PPO</strong>: relies on a learned value function (the critic) and estimates the advantage token by token with GAE: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub><mo>=</mo><msub><mo>∑</mo><mi>l</mi></msub><mo stretchy="false">(</mo><mi>γ</mi><mi>λ</mi><msup><mo stretchy="false">)</mo><mi>l</mi></msup><msub><mi>δ</mi><mrow><mi>t</mi><mo>+</mo><mi>l</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\hat{A}_t = \sum_{l}(\gamma\lambda)^l \delta_{t+l}</annotation></semantics></math></span>, where <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>δ</mi><mi>t</mi></msub><mo>=</mo><msub><mi>r</mi><mi>t</mi></msub><mo>+</mo><mi>γ</mi><mi>V</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>−</mo><mi>V</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)</annotation></semantics></math></span>. If the critic is accurate, the advantage has low variance; if the critic is off, the whole training run goes off with it.</li>
<li><strong>GRPO</strong>: no critic; it samples G responses for the same problem and normalizes with the within-group rewards. In essence it uses the group mean as the baseline—this is REINFORCE&rsquo;s variance-reduction trick, except that the baseline doesn&rsquo;t have to be learned; it is computed directly from the samples in the same group.</li>
</ul>
<p>The GRPO route has three direct consequences, all of which come up later. First, <strong>identical rewards across a group yield all zeros</strong>: when all G responses are right or all are wrong, the mean equals each response&rsquo;s own reward, every advantage is 0, and the problem contributes no gradient at all. Second, <strong>dividing by std carries an implicit weighting</strong>: groups with a small std (say, only 1 of 16 correct) get their weight amplified by the normalization, and the difficulty bias that Dr.GRPO criticizes comes from this term. Third, <strong>the advantage is computed per whole response and shared by every token in it</strong>: in a wrong response, the first 80% of the reasoning steps may well be correct, yet they receive the same negative advantage as the final step where the calculation went wrong—GRPO by itself has no token-level credit assignment.</p>
<p>How the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>min</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\min</annotation></semantics></math></span> behaves depends on the sign of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span>—this is the key to understanding everything:</p>
<ul>
<li><strong><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &gt; 0</annotation></semantics></math></span> (good token, to be reinforced)</strong>: when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>&gt;</mo><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">r &gt; 1+\varepsilon</annotation></semantics></math></span>, the clipped branch is selected, the objective becomes a constant, and <strong>the gradient is zero</strong>. In other words, each update step can raise a token&rsquo;s probability to at most <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">1+\varepsilon</annotation></semantics></math></span> times its value at sampling time; beyond that, no gradient is given.</li>
<li><strong><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&lt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &lt; 0</annotation></semantics></math></span> (bad token, to be suppressed)</strong>: symmetrically, each step can push the probability down at most to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>−</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">1-\varepsilon</annotation></semantics></math></span> times its value at sampling time.</li>
</ul>
<p>There is an easily misread point here that is worth unpacking: <strong>for an advantage of either sign, the clip is one-sided</strong>. Take the case <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &gt; 0</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>&lt;</mo><mn>1</mn><mo>−</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">r &lt; 1-\varepsilon</annotation></semantics></math></span>: the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>min</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\min</annotation></semantics></math></span> still selects <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>⋅</mo><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">r \cdot \hat{A}</annotation></semantics></math></span> (because it is the smaller one). Although the objective&rsquo;s value is small, <strong>the gradient is alive</strong>—<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi><mi>J</mi><mi mathvariant="normal">/</mi><mi>d</mi><mi>π</mi><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">dJ/d\pi &gt; 0</annotation></semantics></math></span>, and the optimizer still pulls this good token&rsquo;s probability back up with the full slope <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span>. A small value does not mean suppression: <strong>the value is the outcome; the gradient is the action</strong>. What the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>min</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\min</annotation></semantics></math></span> does is <strong>withdraw an incentive</strong>, never <strong>apply a reverse incentive</strong>. So the complete structure is: for good tokens, only the incentive to rise is capped (the plateau on the right); for bad tokens, only the incentive to suppress is capped (the plateau on the left); correction in the opposite direction always goes through with the full gradient.</p>
<p>The core mechanism chart in the interactive report is drawn on exactly this logic. How to read it: the horizontal axis is the ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span>, the vertical axis is this token&rsquo;s contribution to the objective, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">J(r)</annotation></semantics></math></span>, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span> is normalized to 1. For <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>&lt;</mo><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">r &lt; 1+\varepsilon</annotation></semantics></math></span>, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo><mo>=</mo><mi>r</mi></mrow><annotation encoding="application/x-tex">J(r)=r</annotation></semantics></math></span> is a rising straight line (the gradient is alive); once <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>&gt;</mo><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">r &gt; 1+\varepsilon</annotation></semantics></math></span>, it becomes a horizontal plateau (the gradient stops there).</p>
<p>Chapter 6 of the RLHF Book also makes a point of noting that the clip is not always at work. Only when the ratio actually goes out of bounds and the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>min</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\min</annotation></semantics></math></span> happens to select the clipped branch is the gradient cut off. The clip fraction is the metric that monitors exactly this. Keep this in mind; it will come in handy later.</p>
<h2 id="symmetric-clipping-asymmetric-effects">Symmetric Clipping, Asymmetric Effects</h2>
<p>Now for some arithmetic. Let <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi><mo>=</mo><mn>0.2</mn></mrow><annotation encoding="application/x-tex">\varepsilon = 0.2</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &gt; 0</annotation></semantics></math></span>, and consider two tokens:</p>
<ul>
<li>A token with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo>=</mo><mn>0.9</mn></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}} = 0.9</annotation></semantics></math></span>: the upper bound on its probability is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.9</mn><mo>×</mo><mn>1.2</mn><mo>=</mo><mn>1.08</mn></mrow><annotation encoding="application/x-tex">0.9 \times 1.2 = 1.08</annotation></semantics></math></span>. A probability cannot exceed 1, so <strong>it is effectively unconstrained</strong>—in one round it can be pushed from 0.9 to above 0.99.</li>
<li>A token with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo>=</mo><mn>0.01</mn></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}} = 0.01</annotation></semantics></math></span>: its upper bound is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.01</mn><mo>×</mo><mn>1.2</mn><mo>=</mo><mn>0.012</mn></mrow><annotation encoding="application/x-tex">0.01 \times 1.2 = 0.012</annotation></semantics></math></span>. <strong>An absolute increment of 0.002</strong>—it barely moves.</li>
</ul>
<p>First, let me pin down &ldquo;one round&rdquo; precisely: it means one &ldquo;sample → update&rdquo; cycle. After sampling, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}}</annotation></semantics></math></span> is frozen as a snapshot, and then multiple gradient updates are run on this batch of data (16 for DAPO, paper §4.1); during this time <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_\theta</annotation></semantics></math></span> gradually drifts away from the snapshot and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span> starts drifting from 1. Once <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span> crosses <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">1+\varepsilon</annotation></semantics></math></span>, that token&rsquo;s gradient is cut off, and the remaining updates in this round no longer push it. So the precise meaning of &ldquo;an upper bound of 0.012&rdquo; is <strong>the point where the incentive vanishes</strong>, not a hard clamp—with a large step, the probability may overshoot it in a single step; it&rsquo;s just that once past it, the incentive drops to zero: nothing pushes it further, and nothing pulls it back.</p>
<p>And the budget <strong>refreshes every round</strong>: the next round resamples, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}}</annotation></semantics></math></span> is updated, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span> resets to 1, and the token gets another <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>×</mo><mn>1.2</mn></mrow><annotation encoding="application/x-tex">\times 1.2</annotation></semantics></math></span> budget. So a low-probability token grows along a multiplicative path: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.01</mn><mo>→</mo><mn>0.012</mn><mo>→</mo><mn>0.0144</mn><mo>→</mo><mo>⋯</mo></mrow><annotation encoding="application/x-tex">0.01 \to 0.012 \to 0.0144 \to \cdots</annotation></semantics></math></span>—exponential growth in ratio terms, yet in absolute terms it climbs very slowly through the first stretch; a high-probability token hits 1.0 and saturates within a round or two. The suppression direction is symmetric but harsher: with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&lt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &lt; 0</annotation></semantics></math></span>, each round can cut the probability down to as little as <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>×</mo><mn>0.8</mn></mrow><annotation encoding="application/x-tex">\times 0.8</annotation></semantics></math></span>, leaving 0.33 after five rounds, so low-probability tokens are quickly beaten down to near 0—<strong>multiplicative decay on the way down is much faster than multiplicative growth on the way up</strong>.</p>
<p>This is exactly where the problem lies: <strong>clipping is measured on the &ldquo;ratio&rdquo; scale, but learning takes effect on the &ldquo;probability&rdquo; axis.</strong> With the same 20% tolerance on the ratio, a high-probability token gets close to 0.2 of room in absolute probability, while a low-probability token gets 0.002. The ranges of movement of the two kinds of token differ by nearly a hundredfold.</p>
<p>And &ldquo;exploration&rdquo; in RL is carried precisely by low-probability tokens—tokens the current policy is not inclined to pick but that may lead to new solutions. DAPO&rsquo;s measurements bear this out: the tokens actually clipped by the upper bound during training have an average probability below 0.2 (Figure 3a of the paper). In other words, <strong>the tokens the clipping mechanism keeps pinning down are precisely the exploration tokens</strong>; exploitation tokens (tokens the model is already very sure about) are almost never constrained.</p>
<p>Long-CoT settings amplify this bias: a reasoning chain runs to thousands of tokens, an exploration token at a critical turning point can move only 0.2% per round, and before it has a chance to grow, the rest of the distribution has already sharpened further.</p>
<h2 id="the-closed-loop-of-collapse">The Closed Loop of Collapse</h2>
<p>Put this micro-level bias into GRPO&rsquo;s training dynamics and you get a complete failure chain:</p>
<ol>
<li>Exploration tokens&rsquo; probabilities can&rsquo;t grow, high-probability tokens keep climbing, the distribution sharpens, and <strong>entropy drops</strong>;</li>
<li>A sharper distribution means the responses sampled as a group (say, 16) for the same problem become <strong>more and more alike</strong>;</li>
<li>GRPO&rsquo;s advantage is computed by within-group normalization; as within-group rewards converge, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>std</mtext><mo>→</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\text{std} \to 0</annotation></semantics></math></span> and all advantages approach zero;</li>
<li><strong>The gradient signal vanishes</strong>; the model loses its direction of improvement and can only keep reinforcing patterns it is already sure of—back to step 1.</li>
</ol>
<p>In the RLVR setting this loop is especially lethal, because its entire training signal is the within-group reward variance. The reward is a rule-based 0/1 (+1 for a correct answer, -1 for a wrong one), with no reward model to provide a continuous signal. Once diversity dries up, the signal source is cut off. Entropy collapse is not &ldquo;a monitoring metric that looks bad&rdquo;; it is training burning its own fuel.</p>
<p>This also explains the logic behind the ordering of DAPO&rsquo;s four techniques: Clip-Higher prevents the collapse, while Dynamic Sampling (filtering out prompts whose samples are all correct or all wrong) stanches the bleeding once the collapse has already produced zero-gradient groups. One treats the root cause, the other the symptoms; the paper puts Clip-Higher in the first section.</p>
<h2 id="clip-higher-raising-only-the-upper-bound-is-deliberate">Clip-Higher: Raising Only the Upper Bound Is Deliberate</h2>
<p>What Clip-Higher does is split the symmetric clip into two hyperparameters:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>clip</mtext><mrow><mo fence="true">(</mo><msub><mi>r</mi><mi>t</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo separator="true">,</mo><mtext> </mtext><mn>1</mn><mo>−</mo><msub><mi>ε</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub><mo separator="true">,</mo><mtext> </mtext><mn>1</mn><mo>+</mo><msub><mi>ε</mi><mrow><mi>h</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub><mo fence="true">)</mo></mrow><mo separator="true">,</mo><mspace width="1em"/><msub><mi>ε</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub><mo>=</mo><mn>0.2</mn><mo separator="true">,</mo><mtext> </mtext><msub><mi>ε</mi><mrow><mi>h</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub><mo>=</mo><mn>0.28</mn></mrow><annotation encoding="application/x-tex">\text{clip}\left(r_t(\theta),\ 1-\varepsilon_{low},\ 1+\varepsilon_{high}\right),\quad \varepsilon_{low}=0.2,\ \varepsilon_{high}=0.28</annotation></semantics></math></span><p>The upper bound goes from 1.2 to 1.28, so a low-probability exploration token&rsquo;s room to rise in a single round goes from +20% to +28%; compounded by the multiplicative accumulation of the budget across rounds, the effect is considerable.</p>
<p>The question worth asking is: why not touch the lower bound? The paper is blunt about it—<strong>if <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ε</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\varepsilon_{low}</annotation></semantics></math></span> were loosened (say, allowing probabilities to be pushed down to 0.6×), the gradient from negative advantages would, by the same ratio logic, push low-probability tokens straight down to near 0</strong>; those tokens would effectively be deleted from the sampling space, and the collapse would only come faster. We did the multiplicative-decay math earlier: the downward direction is already faster than the upward one, so clipping in the suppression direction should, if anything, be tightened, not loosened.</p>
<p>So Clip-Higher is not some vague move like &ldquo;opening the door to exploration a little wider&rdquo;; it is a precise piece of asymmetric surgery: <strong>loosen the restraints on the way up, stay on guard on the way down</strong>.</p>
<p>In terms of results, Figure 2 of the paper shows that after Clip-Higher is applied, the entropy curve clearly recovers and AIME accuracy rises in step. The ablation chain (Qwen2.5-32B, AIME 2024 avg@32):</p>
<table>
  <thead>
      <tr>
          <th>Configuration</th>
          <th>AIME score</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Naive GRPO</td>
          <td>30</td>
      </tr>
      <tr>
          <td>+ Overlong Filtering</td>
          <td>36</td>
      </tr>
      <tr>
          <td>+ Clip-Higher</td>
          <td>38</td>
      </tr>
      <tr>
          <td>+ Soft Overlong Punishment</td>
          <td>41</td>
      </tr>
      <tr>
          <td>+ Token-level Loss</td>
          <td>42</td>
      </tr>
      <tr>
          <td>+ Dynamic Sampling (full DAPO)</td>
          <td>50</td>
      </tr>
  </tbody>
</table>
<p>To be honest: Clip-Higher on its own contributes only +2 points, not the largest gain in the chain (Overlong Filtering gives +6, and adding Dynamic Sampling at the end gives +8). But its role isn&rsquo;t to add points—what it fixes is the stall point of &ldquo;whether training can keep scaling&rdquo;. Once entropy collapses, the three techniques that follow have nothing to work with.</p>
<h2 id="two-premises-rarely-spelled-out">Two Premises Rarely Spelled Out</h2>
<p><strong>Premise one: the clip activates only when training is off-policy.</strong> Chapter 6 of the RLHF Book points out that in RL practice for language models, many algorithms take only one gradient step per batch—in that case the ratio is identically 1, the clip never triggers, and what actually acts as the regularizer is the KL penalty. DAPO, by contrast, takes 16 gradient steps per rollout (mini-batch 512, paper §4.1): the policy is updated multiple times within a batch, and only then does the ratio deviate from 1 and the clip actually start working. <strong>So the value of the Clip-Higher knob is coupled to how off-policy the training is</strong>: if your training is strictly single-step on-policy, this knob simply does not exist. Conversely, asynchronous RL (where rollout and training overlap and samples are inherently off-policy) is the soil in which it really comes into play.</p>
<p><strong>Premise two: DAPO also drops the KL term.</strong> First, let me be clear about the KL penalty itself. The KL divergence measures how far the current policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_\theta</annotation></semantics></math></span> is from the reference policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mrow><mi>r</mi><mi>e</mi><mi>f</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\pi_{ref}</annotation></semantics></math></span> (usually the frozen SFT/base model):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>D</mi><mrow><mi>K</mi><mi>L</mi></mrow></msub><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mi mathvariant="normal">∥</mi><msub><mi>π</mi><mrow><mi>r</mi><mi>e</mi><mi>f</mi></mrow></msub><mo stretchy="false">)</mo><mo>=</mo><msub><mi mathvariant="double-struck">E</mi><mrow><mi>x</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></msub><mrow><mo fence="true">[</mo><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>−</mo><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mrow><mi>r</mi><mi>e</mi><mi>f</mi></mrow></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">D_{KL}(\pi_\theta \| \pi_{ref}) = \mathbb{E}_{x \sim \pi_\theta}\left[\log \pi_\theta(x) - \log \pi_{ref}(x)\right]</annotation></semantics></math></span><p>Classic RLHF puts it into the objective as a penalty term (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>−</mo><mi>β</mi><msub><mi>D</mi><mrow><mi>K</mi><mi>L</mi></mrow></msub></mrow><annotation encoding="application/x-tex">-\beta D_{KL}</annotation></semantics></math></span>), for two reasons: first, <strong>to prevent reward hacking</strong>—the reward model is a leaky proxy, and a policy that strays far enough can find outputs the RM has never seen and blindly scores high; the KL tethers the search to the region of the distribution the RM has seen; second, to preserve the initial model&rsquo;s language abilities and keep formatting from breaking down. The engineering cost is that a frozen reference model has to stay resident in GPU memory, and every token needs a forward pass through both models.</p>
<p>DAPO removes it entirely (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>β</mi><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\beta=0</annotation></semantics></math></span>, so the reference model doesn&rsquo;t need to be loaded), with the reasoning given in §2.3 of the paper: the KL penalty was originally meant to &ldquo;align, but don&rsquo;t stray too far from the initial model&rdquo;, whereas the goal of long-CoT training is <strong>precisely to depart substantially from the initial model</strong>—the base model cannot write long chains of thought or reflect on itself to begin with; these behaviors depend entirely on RL reshaping the distribution, and staying tethered to the anchor amounts to not letting the ship sail. There is also an unstated enabling condition: RLVR rewards are rule-based judgments, so <strong>there is no RM to hack</strong>, and the anti-hacking half of the KL&rsquo;s motivation simply disappears.</p>
<p>Putting the two premises together reveals an implicit gear shift in DAPO: in the classic setup, drift is held in check by the KL—an <strong>absolute anchor</strong> that forbids straying too far from the initial model; once DAPO drops the KL, only the clip remains—a <strong>relative anchor</strong> that forbids each step from straying too far from the previous one but allows the policy to keep moving farther away. The trust-region role has passed from one anchor to the other. Note also that dropping the KL and raising the upper bound point in the same direction—both loosen the restraints on exploration: the KL penalty pulls the distribution toward the reference model token by token, and only with it gone does the policy dare to venture into new regions. The paper does not say outright that the two were designed to work together—this is my inference; but judged by the goal of &ldquo;preventing collapse&rdquo;, they are indeed fighting in the same trench. Of course, removing the anchor is not free: when training spins out of control, nothing pulls the model back, which is why DAPO relies so heavily on its monitoring dashboard (entropy, length, mean probability), and why it needs substitute stabilizers such as the clip and dynamic sampling.</p>
<h2 id="cross-checking-the-implementations">Cross-Checking the Implementations</h2>
<p>Beyond the mechanism on paper, I checked three implementations, and the details largely line up:</p>
<ul>
<li><strong>rlhf-book</strong>&rsquo;s <code>DAPOLoss</code> in <code>code/policy_gradients/loss.py</code>: decoupled clip (<code>clip_eps_lo=0.2 / clip_eps_hi=0.28</code>, see <code>configs/dapo.yaml</code>), token-level loss normalization (the loss is summed and then divided by the total number of valid tokens across the whole batch, rather than GRPO&rsquo;s original per-sequence mean followed by a mean over sequences), and no KL. The teaching implementation is consistent with the paper.</li>
<li><strong>verl&rsquo;s production implementation</strong> (<code>verl/trainer/ppo/core_algos.py</code>): <code>clip_ratio_low</code> / <code>clip_ratio_high</code> are independent config options that fall back to the symmetric <code>clip_ratio</code> when unset—that is, DAPO&rsquo;s asymmetric clipping is already a first-class citizen, but the default behavior is still classic PPO. The file also contains dual-clip&rsquo;s <code>clip_ratio_c=3.0</code>, which guards against ratio blow-up on negative-advantage samples—a problem in a different direction from the one Clip-Higher solves.</li>
<li><strong>Industry corroboration</strong>: MiniMax&rsquo;s CISPO likewise allows asymmetric clipping bounds; VAPO adopts clip-higher directly. This insight is no longer unique to DAPO; it has become an engineering consensus in long-CoT RL.</li>
</ul>
<p>Let me also mention a fact-checking case: the Chinese-language course hands-on-modern-rl, in its explanation of Clip-Higher, says &ldquo;a lower bound of 0.8 means a low-probability action can be pushed down to 0.008 at most, almost completely suppressed&rdquo;, conflating the suppressive effect of the lower bound with the room to rise under the upper bound, which can easily lead readers to think it is the lower bound that was changed. The paper&rsquo;s actual mechanism is exactly the opposite: what changes is the upper bound; the lower bound stays at 0.2 precisely because loosening it further would crush low-probability tokens. The course states that it was generated with AI assistance and has not yet been fully reviewed; when reading secondhand material, always check back against the original—this very check turned up one such error.</p>
<h2 id="addendum-rlvr-and-grpo-are-not-on-the-same-layer">Addendum: RLVR and GRPO Are Not on the Same Layer</h2>
<p>When reading this post, it is easy to say RLVR and GRPO in one breath, but they are actually two orthogonal axes:</p>
<table>
  <thead>
      <tr>
          <th></th>
          <th>Question it answers</th>
          <th>Which layer</th>
          <th>Other options on the same layer</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>RLVR</strong></td>
          <td>Where does the reward come from?</td>
          <td>Reward source / training paradigm</td>
          <td>Human preferences (RLHF&rsquo;s starting point), learned reward models, verifiable rule-based rewards</td>
      </tr>
      <tr>
          <td><strong>GRPO</strong></td>
          <td>Given the reward, how do you update the model?</td>
          <td>Policy optimization algorithm</td>
          <td>REINFORCE, PPO, DAPO, GSPO</td>
      </tr>
  </tbody>
</table>
<p>The two can be combined freely: RLVR + PPO works perfectly well (VAPO is exactly the value-based route running on verifiable rewards); RM + GRPO holds up too (when GRPO came out of DeepSeekMath it was not tied to rule-based rewards, and within-group normalization works for any scalar reward). RLVR + GRPO became the mainstream recipe because the two bring out the best in each other: RLVR&rsquo;s rule-based rewards are cheap, objective, and robust to reward hacking, but they are a sparse 0/1 signal; GRPO&rsquo;s within-group normalization happens to suit 0/1 well—sample a group for the same problem and the ratio of right to wrong naturally forms a baseline, with no need to train a separate critic to estimate &ldquo;how many points this line of reasoning is worth&rdquo;. Conversely, PPO&rsquo;s critic is hard to train accurately on sparse 0/1 rewards over sequences thousands of tokens long.</p>
<p>Placed back into this post&rsquo;s failure chain, the division of labor between the two is clear as well: RLVR determines <strong>how fragile the signal source is</strong> (0/1 rewards come in only three patterns—all correct, all wrong, or a mix—and the first two give zero gradient outright), while GRPO determines <strong>what the signal depends on</strong> (within-group reward variance, i.e., sampling diversity). All four of DAPO&rsquo;s techniques are patches at GRPO&rsquo;s algorithm layer and leave RLVR&rsquo;s reward source untouched; improvements in the other direction—building more verifiable tasks for RLVR, refining rewards down to the process level (PRM)—are surgery at the reward-paradigm layer, a path orthogonal to the algorithm-layer patches.</p>
<h2 id="my-assessment-and-open-questions">My Assessment and Open Questions</h2>
<p><strong>Assessment</strong>: At its core, Clip-Higher turns the &ldquo;pace of exploration&rdquo; from an implicit default left over from history (ε=0.2, from the continuous-control settings of 2017) into an explicit knob geared to long-CoT reasoning. The 0.28 is not theoretically derived; it is an empirical value, but the diagnostic framework behind it—the asymmetry of ratio clipping on the probability axis—has general explanatory power. From now on, whenever you see &ldquo;training was going fine, then entropy suddenly collapsed&rdquo;, first checking the probability distribution of the tokens held down by the clip should become standard practice.</p>
<p><strong>Open questions</strong>, in order of how much I care about them:</p>
<ol>
<li><strong>Entropy has no target range.</strong> The paper&rsquo;s empirical conclusion is that &ldquo;a slow rise in entropy is beneficial&rdquo;, but it also acknowledges that excessively high entropy corresponds to gibberish and repetition. So should ε_high be adjusted adaptively along with the entropy curve rather than fixed at 0.28? I haven&rsquo;t seen anyone study this systematically.</li>
<li><strong>Order effects in the ablation.</strong> Table 1 is a cumulative chain: Clip-Higher is tested after Overlong Filtering, so its +2 points were measured on a base that already includes earlier changes. What its clean, standalone contribution is remains unknown.</li>
<li><strong>Compatibility with sequence-level clipping.</strong> GSPO lifts the importance sampling ratio to the sequence level (to fix the numerical instability of token-level ratios in MoE models); does the logic of &ldquo;raising the token-level upper bound&rdquo; still hold then? The <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">GSPO post</a>
 unpacks exactly this question.</li>
<li><strong>Robustness under asynchronous training.</strong> In asynchronous RL the ratio distribution drifts further and is more skewed; whether a fixed 0.28 is appropriate across different degrees of off-policyness is not covered by the paper.</li>
</ol>
<h2 id="references">References</h2>
<ul>
<li>DAPO paper: <a href="https://arxiv.org/abs/2503.14476">arXiv:2503.14476</a>
 (primary source for the mechanism, numerical examples, ablation data, and training details)</li>
<li>RLHF Book Chapter 6 and its companion implementation: <a href="https://github.com/natolambert/rlhf-book/blob/main/code/policy_gradients/loss.py"><code>code/policy_gradients/loss.py</code></a>
, <a href="https://github.com/natolambert/rlhf-book/blob/main/code/policy_gradients/configs/dapo.yaml"><code>code/policy_gradients/configs/dapo.yaml</code></a>
 (<a href="https://github.com/natolambert/rlhf-book">natolambert/rlhf-book</a>
)</li>
<li>verl production implementation: <a href="https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py"><code>verl/trainer/ppo/core_algos.py</code></a>
 (clip_ratio_low/high, dual-clip)</li>
<li>Bilibili video: <a href="https://www.bilibili.com/video/BV1LgXbY5EFD">Analysis of the GRPO loss/objective and its biases (DAPO, Dr. GRPO), BV1LgXbY5EFD</a>
 (note: this episode mainly covers dynamic sampling and length bias, and Dr. GRPO&rsquo;s standard-deviation issue; it spends very little time on Clip-Higher)</li>
<li>hands-on-modern-rl <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter18_grpo/deepseek-dapo.md"><code>docs/chapter18_grpo/deepseek-dapo.md</code></a>
 (a Chinese-language explainer; the wording in its Clip-Higher section is muddled, as flagged in the main text)</li>
<li>Notion &ldquo;Common Language&rdquo; notes page (the three rounds of industrial reworking of GRPO: DAPO / Dr.GRPO / GSPO)</li>
</ul>
<hr>
<blockquote>
<p>Update log: 2026-08-26, first version published; the same day, additions based on reader Q&amp;A—the definition and caveats of &ldquo;entropy&rdquo;, the two sources of the advantage in PPO and GRPO and their consequences, how to read the one-sided clipping of the min (value vs. gradient), the precise meaning of the &ldquo;single-round budget&rdquo; and its multiplicative accumulation, the layering of RLVR and GRPO, and a detailed look at the KL penalty and the absolute-anchor/relative-anchor gear shift.</p>
</blockquote>
]]></content:encoded>
    </item>
  </channel>
</rss>
