<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>dr-grpo on Njx&#39;Log</title>
    <link>https://njx-njx.github.io/en/tags/dr-grpo/</link>
    <description>Recent content in dr-grpo on Njx&#39;Log</description>
    <image>
      <title>Njx&#39;Log</title>
      <url>https://njx-njx.github.io/blog-cover.jpg</url>
      <link>https://njx-njx.github.io/blog-cover.jpg</link>
    </image>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate>Tue, 27 Oct 2026 15:14:17 +0800</lastBuildDate>
    <atom:link href="https://njx-njx.github.io/en/tags/dr-grpo/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>One Vote or Ten: token-mean vs sample-mean</title>
      <link>https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/</link>
      <pubDate>Sun, 30 Aug 2026 12:07:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/</guid>
      <description>Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.</description>
      <content:encoded><![CDATA[<p>In the same batch, which contributes more to the gradient: a 500-token response or a 5000-token one? GRPO says they count equally, DAPO says the long one counts ten times as much, and both can justify their answer. This is a dispute over loss aggregation: sample-mean gives each response one vote, token-mean gives each token one vote. The DAPO paper (§3.3) says long-CoT settings must switch to token level, but the switch pushes the length skew from the response level to the question level. This post unpacks the three formulas, the two camps&rsquo; arguments, and a tradeoff few people talk about.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/token-mean-vs-sample-mean/">interactive report</a>
 (in Chinese), with a weight calculator that switches among the three vote-counting rules and a side-by-side breakdown of the formulas.</p>
</blockquote>
<h2 id="first-count-the-votes">First, Count the Votes</h2>
<p>The same training batch contains two responses: one is 500 tokens, the other 5000 tokens. How much does each contribute to this step&rsquo;s gradient?</p>
<ul>
<li><strong>GRPO (sample-mean)</strong>: equally. First average over the tokens within each response, then average over the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses — every response, long or short, gets one equal vote;</li>
<li><strong>DAPO (token-mean)</strong>: the long one contributes ten times as much. All tokens in the batch are flattened together and divided by the total token count — one vote per token, so the 5000-token response gets 5000 votes;</li>
<li><strong>Dr.GRPO</strong>: formally also one vote per response, but the divisor changes from &ldquo;each response&rsquo;s actual length&rdquo; to a global constant, which changes what a vote means (that is the topic of the <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">Dr.GRPO post</a>
).</li>
</ul>
<p>All three answers are backed by papers, and all three camps are working on <strong>the same spot in the same expression</strong> — the loss&rsquo;s aggregation divisor. This knob is orthogonal to the advantage normalization (the std half) covered in the Dr.GRPO post: that post&rsquo;s question was &ldquo;how is the advantage computed?&rdquo;; this post&rsquo;s question is &ldquo;once it is computed, by what rule are the tokens&rsquo; losses added together?&rdquo;</p>
<h2 id="three-formulas-side-by-side">Three Formulas Side by Side</h2>
<p>The GRPO objective (DeepSeekMath&rsquo;s Equation 3, with the KL term omitted):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="script">J</mi><mrow><mi>G</mi><mi>R</mi><mi>P</mi><mi>O</mi></mrow></msub><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><msub><mi mathvariant="normal">ℓ</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo fence="true">]</mo></mrow><mo separator="true">,</mo><mspace width="1em"/><msub><mi mathvariant="normal">ℓ</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo>=</mo><mi>min</mi><mo>⁡</mo><mo fence="false" stretchy="true" minsize="1.2em" maxsize="1.2em">(</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo separator="true">,</mo><mtext> </mtext><mrow><mi mathvariant="normal">c</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">p</mi></mrow><mo stretchy="false">(</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo separator="true">,</mo><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo fence="false" stretchy="true" minsize="1.2em" maxsize="1.2em">)</mo></mrow><annotation encoding="application/x-tex">\mathcal{J}_{GRPO} = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\ell_{i,t}\right], \quad \ell_{i,t}=\min\big(w_{i,t}\hat{A}_i,\ \mathrm{clip}(w_{i,t},1-\varepsilon,1+\varepsilon)\hat{A}_i\big)</annotation></semantics></math></span><p>Here <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>o</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">o_i</annotation></semantics></math></span> is the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span>-th response, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">|o_i|</annotation></semantics></math></span> is its length, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">w_{i,t}</annotation></semantics></math></span> is the probability ratio between the new and old policies for that token, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\hat{A}_i</annotation></semantics></math></span> is the advantage shared by the entire response. Note the structure of the sums: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><msub><mo>∑</mo><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\frac{1}{|o_i|}\sum_t</annotation></semantics></math></span> first averages each response <strong>on its own</strong> into a single number, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mi>G</mi></mfrac><msub><mo>∑</mo><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\frac{1}{G}\sum_i</annotation></semantics></math></span> then averages those <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> numbers. The two responses each cast one vote, regardless of length.</p>
<p>DAPO changes just one spot (Equation 12 of the paper, the part highlighted in red):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="script">J</mi><mrow><mi>D</mi><mi>A</mi><mi>P</mi><mi>O</mi></mrow></msub><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><msub><mi mathvariant="normal">ℓ</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\mathcal{J}_{DAPO} = \mathbb{E}\left[\frac{1}{\sum_{i=1}^{G}|o_i|}\sum_{i=1}^{G}\sum_{t=1}^{|o_i|}\ell_{i,t}\right]</annotation></semantics></math></span><p>The inner <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">\frac{1}{|o_i|}</annotation></semantics></math></span> is gone, the two levels of summation are flattened into one, and the divisor becomes the total token count of the whole group. Every token gets exactly the same weight, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><msub><mo>∑</mo><mi>i</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/\sum_i|o_i|</annotation></semantics></math></span>; a response&rsquo;s total weight is proportional to its length.</p>
<p>The third is Dr.GRPO: the same aggregation structure as DAPO (tokens flattened), but the divisor is a global constant (such as the length cap), and the std is removed from the advantage as well — the Dr.GRPO post went into that in detail; this one looks only at the divisor.</p>
<h2 id="why-dapo-insists-on-switching-two-places-the-penalty-cant-reach">Why DAPO Insists on Switching: Two Places the Penalty Can&rsquo;t Reach</h2>
<p>The argument in §3.3 of the DAPO paper is quite concrete and targets the long-CoT setting. Under sample-mean, long samples &ldquo;contribute disproportionately little to the overall loss&rdquo; (original wording: disproportionately lower contribution), which leads to two bad outcomes:</p>
<p>First, <strong>reasoning patterns in high-quality long samples don&rsquo;t get learned</strong>. In a correct 5000-token chain of reasoning, each token receives a gradient weight of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mi>G</mi><mo>⋅</mo><mn>5000</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">1/(G\cdot 5000)</annotation></semantics></math></span>, only one-tenth of what a token in a 500-token short sample gets. What long samples carry is precisely complex reasoning, so under sample-mean, the more important the content, the lower its per-unit weight.</p>
<p>Second, <strong>bad patterns in overlong samples don&rsquo;t get penalized away</strong>. The paper reports an observation: overly long samples often contain low-quality patterns such as gibberish (garbled text) and repeated words. The penalty on these bad tokens is likewise diluted by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|o_i|</annotation></semantics></math></span> — the longer the sample, the more lightly each bad token is penalized. The result is &ldquo;an unhealthy increase in entropy and response length&rdquo; (the measured curves in Figures 4a/4b of the paper).</p>
<p>After switching to token-mean, a token carries the same weight whether it sits in a long sample or a short one. DAPO is upfront about the effect: as a result, longer sequences &ldquo;have more influence on the overall gradient update than shorter ones&rdquo; (original wording: longer sequences can have more influence). This is not a side effect but a design goal — in the long-CoT era, the gradient should be spent mainly on long responses in the first place.</p>
<h2 id="a-tradeoff-few-people-talk-about">A Tradeoff Few People Talk About</h2>
<p>But the phrase &ldquo;long samples get more votes&rdquo; deserves a pause and a second look: it has two layers of meaning, and DAPO only spells out the first.</p>
<p>The first layer is the response level: within a single response, tokens are no longer diluted by length. This is a direction Dr.GRPO endorses too — the &ldquo;response-level length bias&rdquo; that Dr.GRPO criticizes (when correct, shorter responses are favored; when wrong, longer ones are safer) and the &ldquo;bad tokens don&rsquo;t get penalized&rdquo; problem that DAPO criticizes are two faces of the same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|o_i|</annotation></semantics></math></span>, and the two camps&rsquo; prescriptions reach the same destination by different roads: equal weight per token.</p>
<p>The second layer is the question level: across the whole batch, <strong>questions with long responses take more of the total weight</strong>. A question on which the model&rsquo;s responses average 6000 tokens and one on which they average 600 tokens differ in batch weight by 10x under token-mean; under sample-mean the two are equal. In other words, token-mean pushes the length skew from the response level to the question level — long questions dominate the gradient. In the long-CoT setting this may be exactly what you want (hard, reasoning-heavy questions tend to have long responses too), but it is a real bias: question weights are no longer uniform; instead, each question is overweighted in proportion to its current length. I have not seen this layer of the tradeoff quantified in any paper (this point is an inference; happy to be proven wrong).</p>
<p>There is also a second second-order effect: DAPO&rsquo;s divisor <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">\sum_i|o_i|</annotation></semantics></math></span> is <strong>a random variable that varies from batch to batch</strong>, whereas Dr.GRPO&rsquo;s constant divisor is not. The direct consequence of a random divisor is that the gradient scale fluctuates with the length composition of each batch; the indirect consequence is more interesting — later in training, responses generally get longer, the divisor grows, and each token&rsquo;s effective learning rate decays automatically. In other words, token-mean comes with a built-in implicit learning-rate decay that moves in lockstep with length inflation, a property that neither sample-mean nor a constant divisor has. This is a mechanism-level inference that no paper has measured, but it is a real variable when choosing an aggregation method: what you are choosing is not just a weight allocation but also the dynamics of the gradient scale.</p>
<h2 id="both-say-back-to-ppo-but-not-the-same-ppo">Both Say &ldquo;Back to PPO,&rdquo; but Not the Same PPO</h2>
<p>An easily overlooked historical fact: PPO&rsquo;s original objective was aggregated per token to begin with (summing or averaging over the tokens of the whole batch, with GAE supplying a per-token advantage). <strong>It is GRPO&rsquo;s sample-mean that is the outlier</strong> — because GRPO&rsquo;s advantage is &ldquo;one number per response,&rdquo; averaging each response into one number first and then averaging across responses is semantically the tidiest.</p>
<p>Then DAPO and Dr.GRPO both claim to have &ldquo;restored the PPO objective&rdquo;: DAPO restores <strong>PPO&rsquo;s aggregation scheme</strong> (tokens flattened, but the advantage keeps its z-score, so the std is still there); Dr.GRPO restores <strong>the form of PPO&rsquo;s objective function</strong> (a constant divisor, the advantage reduced to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>−</mo><mtext>mean</mtext></mrow><annotation encoding="application/x-tex">r-\text{mean}</annotation></semantics></math></span>, the std removed). The two knobs — aggregation granularity and advantage normalization — are independent and can be combined freely: in implementations such as verl, the two really are configured separately (from the implementation-level discussion in the Dr.GRPO post: by default, <code>masked_mean</code> in trl/verl adds back the per-response averaging that the formula doesn&rsquo;t have, and auditing whether each divisor &ldquo;is a variable or a constant&rdquo; is the cheapest check there is). This also means that &ldquo;GRPO vs DAPO vs Dr.GRPO&rdquo; is not three algorithms lined up side by side, but a configuration space of at least 2×2, and the good points known so far are all still being found by trial and error along its edges.</p>
<h2 id="which-vote-should-weigh-more-and-when">Which Vote Should Weigh More, and When</h2>
<p>Break it down by scenario and the choice isn&rsquo;t mysterious:</p>
<ul>
<li><strong>Short CoT, low length variance</strong>: the two aggregations are approximately equivalent (lengths are similar, so <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|o_i|</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mo>∑</mo><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/\sum|o_i|</annotation></semantics></math></span> differ by a roughly constant factor), so there&rsquo;s no need to agonize;</li>
<li><strong>Long CoT, bad patterns hiding in long samples</strong> (repetition, gibberish, pointless rambling): token-mean is crucial, otherwise bad tokens will never be penalized enough — this is DAPO&rsquo;s core scenario;</li>
<li><strong>You want uniform weights across questions</strong> (e.g., question difficulty varies widely and you worry about long questions dominating): sample-mean or a constant divisor is more controllable;</li>
<li><strong>You want a stable gradient scale and predictable learning-rate behavior</strong>: a constant divisor (the Dr.GRPO route) is the cleanest; token-mean&rsquo;s random divisor mixes length dynamics into the learning rate.</li>
</ul>
<p>My take: at its core, the aggregation dispute is about &ldquo;in what unit the gradient budget is allocated.&rdquo; Under sample-mean it is allocated per response, implicitly assuming every response carries the same amount of information — true in the era of short answers; under token-mean it is allocated per token, implicitly assuming every token carries the same amount of information — closer to the truth in the long-CoT era, but at the cost of letting length decide the weighting across questions. The paper that is really missing is one that makes this budget allocation <strong>explicit</strong>: weights assigned neither per response nor per token, but according to estimated information content or uncertainty. Until then, DAPO&rsquo;s change is the most pragmatic default for long CoT.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>The effect of token-mean&rsquo;s question-level length skew (cross-question weight ∝ length) on where training ends up has not been publicly quantified;</li>
<li>Nobody has measured how the implicit learning-rate decay from the random divisor interacts with an explicit LR schedule stacked on top of it;</li>
<li>For a full 2×2 ablation of the two knobs, aggregation granularity and advantage normalization (sample/token × with std/without std), the public literature only has the points on the diagonal.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>DAPO paper: <a href="https://arxiv.org/abs/2503.14476">arXiv:2503.14476</a>
; token-level loss in §3.3 (Equation 12), entropy and length curves in Figure 4</li>
<li>Dr.GRPO paper: <a href="https://arxiv.org/abs/2503.20783">arXiv:2503.20783</a>
; the original statements of the response-level length bias and the question-level difficulty bias</li>
<li>DeepSeekMath / GRPO: <a href="https://arxiv.org/abs/2402.03300">arXiv:2402.03300</a>
; the sample-mean objective (Equation 3)</li>
<li>Previous post (covering the advantage-normalization half): <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">Dr.GRPO: Two Normalization Terms, Two Biases</a>
 (it gave the token-mean section number as §3.2 by mistake; the correct one is §3.3, and that has been corrected in passing)</li>
<li>The question-level length skew and the random divisor&rsquo;s implicit LR decay are this post&rsquo;s own inferences, each flagged where it appears in the text</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>“Wrong? Write Longer”: What GRPO Normalizers Skew</title>
      <link>https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/</link>
      <pubDate>Sat, 29 Aug 2026 12:05:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/</guid>
      <description>Part of R1-Zero&amp;rsquo;s growing length may be the optimizer&amp;rsquo;s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.</description>
      <content:encoded><![CDATA[<p>R1-Zero&rsquo;s “length and reward rising together” has been treated as the signature curve of emergent reasoning, but Sea AI Lab&rsquo;s Dr.GRPO paper points out that half of that rise may be the optimizer&rsquo;s own preference: 1/|y| makes the model lean toward writing longer when it answers wrong, and dividing by std lets questions at the difficulty extremes take more of the weight. A mechanistic breakdown of two normalization terms and two biases, plus an implementation-level bias hidden in masked_mean.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/dr-grpo-length-and-difficulty-bias/">interactive report</a>
 (in Chinese), with drill-down worked examples of the biases, a side-by-side comparison of masked_mean implementations, and a decision matrix for choosing a fix.</p>
</blockquote>
<h2 id="an-over-trusted-curve">An Over-Trusted Curve</h2>
<p>The most famous plot from R1-Zero training is the one where response length and training reward rise in lockstep. The community was quick to spin a story around it: the model has learned to think longer, and long CoT keeps taking shape through RL.</p>
<p>Sea AI Lab&rsquo;s Dr.GRPO paper (arXiv:2503.20783) poses a simpler question: on the plateau where reward has stopped rising, why does length keep growing? After pinning down the causes one by one, the paper concludes that this is not the model learning to reason more; rather, <strong>the optimizer keeps pushing the model to generate longer responses</strong>. The driving force comes from two normalization terms in the GRPO objective: they adjust the model&rsquo;s optimization direction in opposite ways for correct and incorrect responses, and their effect on incorrect responses points to one and the same outcome: <strong>when the answer is wrong, the longer the response, the less it gets penalized.</strong></p>
<h2 id="bias-1-1y--per-token-averaging-dilutes-the-penalty">Bias 1: 1/|y| — Per-Token Averaging Dilutes the Penalty</h2>
<p>The GRPO objective (Equation 3 in the paper) contains an inconspicuous denominator:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="script">J</mi><mrow><mi>G</mi><mi>R</mi><mi>P</mi><mi>O</mi></mrow></msub><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><mi>min</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo separator="true">,</mo><mtext> </mtext><mrow><mi mathvariant="normal">c</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">p</mi></mrow><mo stretchy="false">(</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo separator="true">,</mo><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo fence="true">)</mo></mrow><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\mathcal{J}_{GRPO} = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G} \frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\min\left(w_{i,t}\hat{A}_i,\ \mathrm{clip}(w_{i,t},1-\varepsilon,1+\varepsilon)\hat{A}_i\right)\right]</annotation></semantics></math></span><p>What <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> means is this: within a single response, first average the gradient terms over all of its tokens, then add up the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses. In other words, <strong>the gradient weight each token actually receives is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">\hat{A}_i/|y_i|</annotation></semantics></math></span></strong>. A concrete worked example: sample two responses to the same question, one 100 tokens long and one 1000 tokens long, both with an advantage of absolute value 1 (opposite in sign):</p>
<ul>
<li>Correct (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>=</mo><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\hat{A}=+1</annotation></semantics></math></span>): each token of the short response is reinforced by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mn>100</mn><mo>=</mo><mn>0.01</mn></mrow><annotation encoding="application/x-tex">1/100 = 0.01</annotation></semantics></math></span>, while each token of the long response is reinforced by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mn>1000</mn><mo>=</mo><mn>0.001</mn></mrow><annotation encoding="application/x-tex">1/1000 = 0.001</annotation></semantics></math></span>. <strong>Among correct responses, the shorter, the more favored</strong>;</li>
<li>Incorrect (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>=</mo><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\hat{A}=-1</annotation></semantics></math></span>): each token of the short response is penalized by 0.01, while each token of the long response is penalized by only 0.001. <strong>Among incorrect responses, the longer, the less penalized</strong> — a tenfold difference.</li>
</ul>
<p>This is not an anomaly caused by a math error; it is something the objective writes into every update step: correct answers must be concise; wrong answers may as well be long-winded. Late in training, once reward plateaus, the model&rsquo;s room for improvement on the first half is used up, while the second half keeps pushing the model to generate longer responses — incorrect responses in particular keep getting longer. Fig.5 Plot 2 in the paper&rsquo;s Figure 4 clearly shows this phase: GRPO&rsquo;s length curve keeps rising after reward plateaus.</p>
<h2 id="bias-2-dividing-by-std--extreme-difficulty-questions-take-more-weight">Bias 2: Dividing by std — Extreme-Difficulty Questions Take More Weight</h2>
<p>The second normalization term sits in the advantage:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>=</mo><mfrac><mrow><msub><mi>R</mi><mi>i</mi></msub><mo>−</mo><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mo stretchy="false">(</mo><mo stretchy="false">{</mo><msub><mi>R</mi><mi>j</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy="false">)</mo></mrow><mrow><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">d</mi></mrow><mo stretchy="false">(</mo><mo stretchy="false">{</mo><msub><mi>R</mi><mi>j</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">\hat{A}_i = \frac{R_i - \mathrm{mean}(\{R_j\}_{j=1}^G)}{\mathrm{std}(\{R_j\}_{j=1}^G)}</annotation></semantics></math></span><p>Dividing by the within-group reward standard deviation is meant to put advantages on a common scale (a standard RL trick). But the paper points out a key difference: classic RL advantage normalization computes the std over <strong>the whole batch</strong>, with all samples sharing one divisor; GRPO <strong>computes the std separately for each question</strong>, so every question has its own divisor — and different divisors mean different weights.</p>
<p>One more worked example: for question 1, all 8 sampled responses are correct (std≈0.35, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span>≈+2.9 on the z-score scale); for question 2, the 8 samples are four correct and four wrong (std≈0.52, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span>≈±1.9). The more extreme the question — the all-correct giveaway, the all-wrong dead end — the smaller its std, and the larger its weight after the division. The model spends the most update budget on questions it already gets entirely right (the easy ones) and on questions it cannot solve at all (never answered correctly even once), while questions that are “partly right, partly wrong, right in the zone where learning pays off” get the least weight. This is what the paper calls <strong>difficulty bias</strong>.</p>
<p>A caveat on scope: std normalization is not all downside. Its design intent, amplifying the signal in low-variance cases, is to let the model learn something even from questions it gets entirely right. What the paper criticizes is the unequal weighting caused by <strong>per-question normalization</strong>, not normalization itself.</p>
<h2 id="bias-3-hidden-in-the-implementation-not-the-formula">Bias 3: Hidden in the Implementation, Not the Formula</h2>
<p>The third finding may hit engineers harder. The PPO objective (Equation 2 in the paper) <strong>sums</strong> over the response&rsquo;s tokens, with no step that divides by length. But the paper examined mainstream open-source implementations and found that the losses in frameworks such as trl and verl all contain a <code>masked_mean</code> line: it averages using the response&rsquo;s actual token count as the divisor (<code>loss.sum(dim) / mask.sum(dim)</code>). In other words, <strong>the formula has no length bias, but the implementation adds one</strong>.</p>
<p>How did this happen? The paper&rsquo;s conjecture traces it to historical practice: in pretraining, all data is packed into fixed-length contexts, and <code>loss.mean(-1)</code>, averaging over the context length, is both numerically convenient and stable; it is division by a constant and introduces no bias. Once this idiom was carried over to the RL stage, the divisor quietly changed from “the fixed context length” to “the actual length of each response” — from a constant to a variable, and the effect changed with it. Dr.GRPO&rsquo;s implementation fix changes just one line: replace the divisor in <code>masked_mean</code> with a global constant (such as the generation length cap MAX_TOKENS).</p>
<p>The formula&rsquo;s authors and the framework&rsquo;s authors may be running the same set of tests, yet they end up with two different gradients. This kind of “formula–implementation mismatch” is more insidious than paper-level biases: you could read the paper ten times and never catch it; only reading the code reveals it.</p>
<h2 id="drgrpos-fix-and-its-justification">Dr.GRPO&rsquo;s Fix and Its Justification</h2>
<p>The fix is simple: remove both normalizations, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> and std, so that the advantage only subtracts the group mean:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>~</mo></mover><mi>i</mi></msub><mo>=</mo><msub><mi>R</mi><mi>i</mi></msub><mo>−</mo><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mo stretchy="false">(</mo><mo stretchy="false">{</mo><msub><mi>R</mi><mi>j</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\tilde{A}_i = R_i - \mathrm{mean}(\{R_j\}_{j=1}^G)</annotation></semantics></math></span><p>Appendix A of the paper gives three theoretical justifications: with both terms removed, the objective <strong>reverts to the PPO objective (Equation 2)</strong>, and the advantage is just the baseline-free Monte Carlo return plus an unbiased baseline; <strong>subtracting the mean introduces no bias</strong> — the baseline&rsquo;s expected contribution to the gradient is identically zero, a standard result in policy gradients; and the corrected advantage differs from RLOO (Leave-One-Out) only by a constant factor <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mi>G</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">G/(G-1)</annotation></semantics></math></span>, which can be folded into the learning rate and does not affect training dynamics.</p>
<p>The controlled experiments are run in the Oat framework on Qwen2.5-1.5B. The results fall into three layers:</p>
<ul>
<li><strong>Training dynamics</strong>: early on, both GRPO and Dr.GRPO show the R1-Zero-style joint rise of reward and length, but after reward plateaus GRPO&rsquo;s length keeps climbing, while Dr.GRPO&rsquo;s length stabilizes (Fig.5 Plot 1&amp;2);</li>
<li><strong>Behavior</strong>: on benchmarks, the length of Dr.GRPO&rsquo;s incorrect responses drops significantly — the change also mitigates overthinking (Fig.5 Plot 4);</li>
<li><strong>Reproducibility</strong>: after math pretraining, Llama-3.2-3B likewise shows GRPO&rsquo;s length rising together with reward, and Dr.GRPO likewise curbs the continued growth in length (Fig.7). The same recipe also lets the 7B Qwen2.5-Math-7B, using MATH levels 3–5 and training for 27 hours on 8 A100s, reach 43.3% on AIME 2024 — the 7B SOTA at the time, and the whole recipe is extremely simple.</li>
</ul>
<h2 id="a-textbook-erratum-and-the-division-of-labor-with-the-previous-posts">A Textbook Erratum, and the Division of Labor with the Previous Posts</h2>
<p>Section 16.4 of the textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
 also covers this pair of biases, but its wording deserves a correction: it labels “the advantage gets amplified as std approaches zero” as “length bias”, and attributes length inflation to “dividing by std encourages the model to increase within-group variance”. Checked against the original paper, this account is off on two counts: std causes <strong>difficulty bias</strong> (unequal weights across questions), not length bias; and the direct source of length bias is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span>, which has nothing to do with std and acts on the weight of each token within a response. The two normalizations and the two biases each exist independently — the value of the erratum is that choosing a fix depends on this distinction: removing std does not solve the length problem.</p>
<p>The recent posts in this series happen to tackle this at three different levels:</p>
<ul>
<li><strong>DAPO&rsquo;s token-mean</strong> (§3.3): it changes “average within each sample first, then average over the G samples” to “average all tokens in the batch uniformly by the total token count”, i.e. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><msub><mo>∑</mo><mi>i</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><msub><mo>∑</mo><mi>i</mi></msub><msub><mo>∑</mo><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\frac{1}{\sum_i|y_i|}\sum_i\sum_t</annotation></semantics></math></span>. All tokens are thus weighted by the total token count, which formally eliminates length bias — but DAPO&rsquo;s advantage keeps the z-score (including std), so it <strong>only solves half the problem</strong>;</li>
<li><strong>GSPO</strong>: it uses a single ratio at the sequence level and no longer sums over tokens term by term in the objective, so there is no “divide each token by |y|” averaging step at all — it sidesteps this bias structurally;</li>
<li><strong>Dr.GRPO</strong> (this post): it leaves the structure of the objective unchanged and only removes the two variable divisors from the normalization terms.</li>
</ul>
<h2 id="my-three-takeaways">My Three Takeaways</h2>
<p>First, <strong>an RL training curve must be screened for optimizer-induced false signals before anything else</strong>. When “length keeps rising” is cited by academia as headline evidence that “RL is working” and, at the same time, by engineering teams as an anomaly metric that “calls for intervention”, tracing it back to the root matters all the more. The paper also observes that for Qwen2.5-Math models used with a template, the model&rsquo;s pre-RL ability is first damaged by the template and then rebuilt by RL (§3.3) — another class of factor that needs ruling out: part of the “RL gain” is really just ability that was there all along. Before analyzing an RL curve, rule out, one by one, the biases the optimizer introduces and the artifacts the evaluation setup creates.</p>
<p>Second, <strong>Dr.GRPO and dynamic sampling solve different problems; neither replaces the other</strong>. Removing std normalization corrects weight <strong>distortion</strong> (samples that should not be amplified receive too much weight); DAPO&rsquo;s dynamic sampling (questions with std=0 are simply dropped and resampled) deals with <strong>the absence of gradient signal</strong> (when the responses are all correct/all wrong/all zero, the advantage is still 0 after subtracting the mean, and nothing is learned). Note that after removing std, a zero-signal question is still zero-signal — Dr.GRPO brings no gradient to such questions. One ensures correct relative weights within the samples, the other ensures the samples as a whole carry a learnable signal; a real training system needs both configured at once.</p>
<p>Third, <strong>when checking a normalization, first check “whether the divisor is a variable or a constant”</strong>. The three biases here share one pattern: a normalization that looks like routine practice turns out, on closer inspection, to use a denominator that is not a constant — for <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> the divisor is the length of each response, for std it is each question&rsquo;s within-group spread, and for <code>masked_mean</code> it is the mask sum of the current batch. Going through every mean/divide in the training code and asking “can this divisor change from sample to sample?” is the cheapest taste audit there is.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>How the reweighting of “extreme-difficulty questions” after Dr.GRPO removes std plays out in very long training runs (tens of thousands of steps): the time window of the paper&rsquo;s 1.5B/7B experiments is relatively short;</li>
<li>The effective-learning-rate conversion when token-mean (DAPO) and removing the variable divisors (Dr.GRPO) are stacked in the same training stack: there is no public, unified convention;</li>
<li>A systematic check of “how much of the <code>loss.mean(-1)</code> habit from pretraining-era packing still lingers in RL frameworks”: so far there is only the paper&rsquo;s spot check of trl/verl.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Sea AI Lab / NUS: <a href="https://arxiv.org/abs/2503.20783">Understanding R1-Zero-Like Training: A Critical Perspective</a>
 (the source of this post&rsquo;s formulas, experiments, and bias definitions; v2)</li>
<li>Textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
: Section 16.4 (using the corrected wording) and the appendix DAPO cheatsheet (<a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/en/appendix_code_cheatsheet/dapo.md"><code>docs/en/appendix_code_cheatsheet/dapo.md</code></a>
, a side-by-side of DAPO&rsquo;s z-score advantage and the token-mean loss granularity)</li>
<li>RLOO reference: <a href="https://arxiv.org/abs/2402.14740">Back to Basics: Revisiting REINFORCE Style Optimization</a>
</li>
<li>Previous posts: <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
, <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">The Ratio&rsquo;s Wrong Unit: GSPO Goes Sequence-Level</a>
, <a href="https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/">One Model, Two Probabilities: Cost of Mismatch</a>
</li>
</ul>
]]></content:encoded>
    </item>
  </channel>
</rss>
