<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>rl on Njx&#39;Log</title>
    <link>https://njx-njx.github.io/en/tags/rl/</link>
    <description>Recent content in rl on Njx&#39;Log</description>
    <image>
      <title>Njx&#39;Log</title>
      <url>https://njx-njx.github.io/blog-cover.jpg</url>
      <link>https://njx-njx.github.io/blog-cover.jpg</link>
    </image>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate>Tue, 27 Oct 2026 15:14:17 +0800</lastBuildDate>
    <atom:link href="https://njx-njx.github.io/en/tags/rl/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>“The Model Trains Itself”: RSI and Three Loops</title>
      <link>https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/</link>
      <pubDate>Tue, 01 Sep 2026 12:30:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/</guid>
      <description>RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.</description>
      <content:encoded><![CDATA[<p>The term RSI (recursive self-improvement) became hot again in 2025-2026, but the things it refers to often get lumped together. Starting from a practitioner&rsquo;s dichotomy (change the harness but not the weights vs. change the weights directly), this post breaks “self-improvement” into three levels—the data loop, the reward loop, and the system loop—which differ in their preconditions, how far they have actually paid off, and their failure modes.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/rsi-self-improvement-loops/">interactive report</a>
 (in Chinese), with an interactive diagram of the three nested loops, a three-level exam matrix, and a scissors-gap diagram.</p>
</blockquote>
<h2 id="a-conflated-term">A Conflated Term</h2>
<p>&ldquo;RSI&rdquo; (Recursive Self-Improvement) became hot again in 2025–2026, but what people mean by it varies widely: some mean an iterative pipeline that filters rollouts into data and retrains on it, some mean a model rewriting its own training process, and some mean scaling up test-time compute. The term&rsquo;s degree of confusion is directly proportional to the allure of what it promises.</p>
<p>Let me start with a practitioner&rsquo;s framework. In a conversation, Ding Duo, head of post-training at Perplexity, split RSI and continual learning into two routes: <strong>one leaves the model parameters unchanged and optimizes the harness, the loop, and the solving process; the other updates the model weights directly</strong>. His judgment is that the former produces results faster, while the latter may bring deeper changes in capability; and the two routes share three hard problems—controlling cost, avoiding parameter contamination, and proving that what is continually learned is transferable capability. This post will keep coming back to these three problems.</p>
<p>Another common confusion is around the term test-time. Classic TTT (Test-Time Training, Sun et al. 2020) refers to taking gradient updates on the current input at inference time, whereas the &ldquo;test-time scaling&rdquo; popularized by s1 refers to lengthening the reasoning chain and adding budget, without touching the weights. When people in an LLM context talk about a &ldquo;new test-time paradigm&rdquo;, always ask which one they mean—the former actually changes the model; the latter just thinks more. What Ding Duo calls &ldquo;optimizing the harness/loop&rdquo; is in fact a third path: leave the gradients alone and change the scaffolding.</p>
<h2 id="level-one-the-data-loopmost-mature-and-least-like-rsi">Level One: The Data Loop—Most Mature, and Least Like RSI</h2>
<p>The first level is data circulating through the loop: model rollout → filter successful trajectories by verifiable outcomes → mix them into the next round of SFT/RL training → the model gets stronger → roll out again. LoopTool (<a href="https://arxiv.org/abs/2511.09148">arXiv:2511.09148</a>
) has exactly this shape (evaluation uncovers weaknesses → targeted data synthesis → retraining); a more complete industrial example is the WebSailor series (arXiv:2507.02592; the V2 version, 2509.13305, explicitly puts synthetic data + scalable RL in its title): use a strong model to generate browsing trajectories, train with outcome rewards, filter with self-critique, then automatically synthesize harder training tasks. DeepResearcher (2504.03160) reports a phenomenon worth remembering: in this kind of closed-loop training, behaviors such as pre-search planning, information layering, and cross-validation <strong>are not explicitly encouraged by any reward term; they are by-products of RL optimization</strong>.</p>
<p>This level is already industrialized, but strictly speaking it is not &ldquo;self&rdquo;-improvement—the choices in the loop are made by a filter designed by engineers, and the model is just a data source. Its ceiling is also clear: the filtering criteria themselves are hackable (inefficient trajectories that &ldquo;happened to succeed&rdquo; get kept), and the narrower the data distribution, the faster the loop turns, and the faster the model converges itself toward whatever the current verifier likes.</p>
<h2 id="level-two-the-reward-looprlsvr-aims-to-extend-the-self-verification-dividend-to-open-ended-tasks">Level Two: The Reward Loop—RLSVR Aims to Extend the Self-Verification Dividend to Open-Ended Tasks</h2>
<p>The second level is reward circulating through the loop. RLVR&rsquo;s success over the past two years rests on &ldquo;math and code have deterministic verifiers&rdquo;; open-ended tasks (writing, research, dialogue) have no verifier and can only rely on preference models or LLM judges, which brings a triple tax: evaluation bias, the judge-capability bottleneck, and inference cost. RLSVR (arXiv:2607.23802) borrows the concept of a &ldquo;pretext task&rdquo; from self-supervised learning: <strong>apply a task transformation to the open-ended task to turn it into a self-verifiable form</strong>, so that the reward once again comes from the task itself rather than from an external judge.</p>
<p>The conditions for this idea to hold are worth laying out: it requires that the task admit some fidelity-preserving transformation (getting the transformed task right ≈ doing the original task well), and that the verifier be deterministic in the transformed space. The &ldquo;task transformation induces self-verifiable rewards&rdquo; in the paper&rsquo;s title refers to exactly this inducing step. If it holds, open-ended tasks can also tap into the dividends of RLVR&rsquo;s industrialization; if it doesn&rsquo;t—if the transformation loses too much—the self-verifiable reward is optimizing a proxy metric that has come apart from the original task.</p>
<p>What this means for the loops: the reward loop is the most critical of the three levels for RSI. The data loop&rsquo;s filter and the system loop&rsquo;s evaluator both have to answer &ldquo;what counts as getting better&rdquo;, and self-verifiable reward is currently the only answer that does not depend on a continuous supply from humans.</p>
<h2 id="level-three-the-system-looprsi-proper-stuck-on-oversight">Level Three: The System Loop—RSI Proper, Stuck on Oversight</h2>
<p>Only the third level is RSI proper: the model (or what it produces) improves the things used to improve it—better assistants, better data, better evaluations—forming a recursion. Leike et al.&rsquo;s 2018 research agenda for reward modeling (arXiv:1811.07871) already has a section explicitly discussing recursive reward modeling: use AI to help humans evaluate AI outputs, so that evaluation capability grows along with whatever is being evaluated.</p>
<p>The prerequisite for this level is scalable oversight (2211.03540, cited in Chapter 30 of the hands-on-modern-rl textbook as a core direction in alignment): once model capability exceeds human evaluation capability, who verifies that an improvement is real? Every round of the loop makes selections, and once the selection criteria are hackable, <strong>the faster the loop turns, the faster it collapses</strong>—a small-scale version is already visible in the data loop (metric inflation), and the system loop amplifies it to the entire pipeline. This is also why &ldquo;proving that what is continually learned is transferable capability&rdquo; is on Ding Duo&rsquo;s list of hard problems: improvement inside the loop is often &ldquo;getting stronger on this loop&rsquo;s evaluations&rdquo;, which does not necessarily hold outside the loop.</p>
<h2 id="the-three-levels-share-one-exam">The Three Levels Share One Exam</h2>
<p>Putting the three levels together, each of Ding Duo&rsquo;s three hard problems maps onto concrete mechanisms:</p>
<ul>
<li><strong>Cost</strong>: the data loop&rsquo;s cost is rollout and filtering (every iteration samples in full; we did the rollout = prompt × G math earlier in the series); the reward loop&rsquo;s cost is constructing and maintaining the task transformations and verifiers; the system loop&rsquo;s cost is basically impossible to estimate for now—because no instance has yet run the full pipeline end to end.</li>
<li><strong>Parameter contamination</strong>: only level three is involved. Continual gradient updates let new knowledge interfere with old capabilities, and the contamination accumulates with the number of iterations—this is one reason the &ldquo;don&rsquo;t change the weights&rdquo; route is currently underrated: optimizing the harness and loop doesn&rsquo;t contaminate parameters, and the gains are immediate; engineering work like DeepSeek V3.2 (self-added verifiers, switching the sampler to mHC) is this route paying off.</li>
<li><strong>Transferability</strong>: none of the three levels has a good evaluation protocol. Evaluation inside a loop is contaminated by the loop itself (the training target and the evaluation target share the same source), and cross-loop evaluation is too expensive.</li>
</ul>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>the &ldquo;don&rsquo;t change the weights&rdquo; route is currently the main way RSI actually pays off, and theorists underrate it</strong>. Scaffold optimization (tools, context engineering, verifier internalization, sampling procedures) doesn&rsquo;t change parameters, doesn&rsquo;t contaminate weights, and can be rolled back, and at the current stage its return per unit of compute is higher than that of changing the weights directly. Expelling it from the ranks of &ldquo;self-improvement&rdquo; is a misallocation of resources caused by terminological purism.</p>
<p>Second, <strong>the three levels are bottlenecked in different places</strong>. The data loop is stuck on the hackability of its filter; the reward loop on the fidelity of the task transformation; the system loop on oversight (without a solution to scalable oversight, recursion can only be recursive self-deception). Discuss &ldquo;RSI progress&rdquo; without specifying the level, and the conclusions are bound to be vague.</p>
<p>Third, <strong>loop speed is a double-edged sword, and an amplifier of safety problems</strong>. Every paper on self-improvement talks about convergence speed, and few talk about runaway speed at the same time; yet the two share the same structure—the loop compounds &ldquo;small selection biases&rdquo;. For self-improving systems, what should be monitored is the <strong>scissors gap</strong> between in-loop and out-of-loop metrics, not any single metric.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>Whether there is a systematic way to construct task transformations for open-ended tasks (rather than hand-tuning them task by task); RLSVR only provides the paradigm;</li>
<li>Where the capability ceiling of the &ldquo;don&rsquo;t change the weights&rdquo; route lies—whether there are capabilities that can only be obtained by changing the weights; there is no clean evidence yet;</li>
<li>An evaluation protocol for cross-loop transferability: who evaluates, with what data, and how to keep the evaluation from being contaminated by any one of the loops.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Notes on a conversation with Ding Duo: 《在模型持续进步的时代，创业者应该站在哪里》§20 (the two routes for RSI and continual learning, and the three hard problems)</li>
<li>RLSVR: <a href="https://arxiv.org/abs/2607.23802">arXiv:2607.23802</a>
 (abstract and core claims)</li>
<li>WebSailor: <a href="https://arxiv.org/abs/2507.02592">arXiv:2507.02592</a>
, <a href="https://arxiv.org/abs/2509.13305">V2 2509.13305</a>
 (titles and positioning within the series); DeepResearcher <a href="https://arxiv.org/abs/2504.03160">arXiv:2504.03160</a>
</li>
<li>Recursive reward modeling: Leike et al. <a href="https://arxiv.org/abs/1811.07871">arXiv:1811.07871</a>
; Scalable Oversight <a href="https://arxiv.org/abs/2211.03540">arXiv:2211.03540</a>
 (hands-on-modern-rl textbook: <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter30_alignment_failures/scaling-and-defenses.md"><code>docs/chapter30_alignment_failures/scaling-and-defenses.md</code></a>
)</li>
<li>Classic Test-Time Training: Sun et al. <a href="https://arxiv.org/abs/1909.13231">arXiv:1909.13231</a>
; test-time scaling: s1 <a href="https://arxiv.org/abs/2501.19393">arXiv:2501.19393</a>
</li>
<li>Previous post: <a href="https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/">Who&rsquo;s to Blame When a 7-Turn Interaction Fails: Reward Credit Assignment for Multi-Turn Trajectories</a>
 (previous post: within-episode credit assignment; this post: cross-episode loops)</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>A 7-Turn Failure: Assigning Blame in GRPO/GSPO</title>
      <link>https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/</link>
      <pubDate>Mon, 31 Aug 2026 12:55:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/</guid>
      <description>Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token&amp;rsquo;s opening.</description>
      <content:encoded><![CDATA[<p>In single-turn RL the whole sequence shares one advantage, and nobody has ever lost sleep over it; in multi-turn agentic RL, the same trajectory can get the turn-1 search right, send the turn-2 query off course, and end with a wrong answer—so how to split a single trajectory reward becomes a real design decision. This post unpacks the formalism (action mask, trajectory probability), the credit-assignment spectrum (ORM/PRM/turn discounting/three middle-ground routes), and the opening GSPO-token leaves for multi-turn training.</p>
<blockquote>
<p>This post comes with a companion <a href="https://njx-njx.github.io/reports/turn-level-reward-credit-assignment/">interactive report</a>
 (in Chinese), featuring a toggleable ORM/PRM credit-assignment comparison, backward discounting with a γ slider, and a breakdown of the GSPO-token formula.</p>
</blockquote>
<h2 id="a-problem-that-doesnt-exist-in-single-turn-training">A Problem That Doesn&rsquo;t Exist in Single-Turn Training</h2>
<p>Consider a five-turn trajectory: turn 1&rsquo;s search hits the key evidence; turn 2 drifts off course while rewriting the query and brings back irrelevant content; turn 3 corrects course and gets halfway back; turn 4 makes a calculation error; turn 5 gives a wrong answer. The environment returns just one number: failure, reward = 0.</p>
<p>Single-turn RL doesn&rsquo;t have this problem—the entire response shares one advantage, and &ldquo;all right or all wrong&rdquo; is taken for granted. In the multi-turn setting this crude logic breaks down immediately: turn 1 clearly did the right thing, so why should it be penalized along with turn 4? On top of that there is a further engineering layer: the trajectory mixes two kinds of tokens, those the model generated itself (thinking, tool-call arguments, the final answer) and those returned by the environment (search results, code output), and the latter should never enter the gradient at all. So &ldquo;how does turn-level reward get into GRPO/GSPO&rdquo; is really three questions rolled into one: <strong>which tokens count as actions, how much advantage each turn gets, and which tokens that advantage gets multiplied onto</strong>. This post takes them apart in that order.</p>
<h2 id="first-pin-down-the-object-trajectory-probability-and-the-action-mask">First, Pin Down the Object: Trajectory Probability and the Action Mask</h2>
<p>In the formalization of multi-turn trajectories (the textbook adopts the POMDP formulation from the AppWorld paper, arXiv:2504.11536), the full state is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>t</mi></msub><mo>=</mo><mo stretchy="false">[</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>c</mi><mo separator="true">,</mo><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi></mrow></msub><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">z_t = [s_0, c, x_{1:t}]</annotation></semantics></math></span>: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">s_0</annotation></semantics></math></span> is the hidden initial environment state (database snapshot, REPL state), <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span> is the task context, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">x_{1:t}</annotation></semantics></math></span> is the entire token history so far—both what the model generated and what the environment returned. The model sees only the text part, which is why it is called partially observable.</p>
<p>The chain-rule factorization of the trajectory probability has one key detail:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>ρ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo>∣</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>c</mi><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="double-struck">I</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>x</mi><mo stretchy="false">)</mo><munder><mo>∏</mo><mrow><mi>t</mi><mo>∈</mo><mi>a</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow></munder><msub><mi>p</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>x</mi><mi>t</mi></msub><mo>∣</mo><mi>c</mi><mo separator="true">,</mo><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\rho_\theta(x \mid s_0, c) = \mathbb{I}(s_0, x) \prod_{t \in a(x)} p_\theta(x_t \mid c, x_{1:t-1})</annotation></semantics></math></span><p>The product runs <strong>only over <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a(x)</annotation></semantics></math></span>, the set of positions of model-generated tokens</strong>; the observation tokens returned by the environment are determined by the environment dynamics and are not part of the product. Translated into the loss, this is the action mask: a 0/1 vector the same length as the trajectory, with model tokens marked 1 and the prompt and tool returns marked 0. SearchR1&rsquo;s implementation makes this very explicit: the contents of <code>&lt;think&gt;</code> and <code>&lt;search&gt;</code> take part in training, while the <code>&lt;information&gt;</code> segments returned by the retriever are masked out entirely. Letting environment tokens into the gradient as well would amount to forcing the model to &ldquo;learn to predict which web pages the search engine returns&rdquo;, contaminating the policy gradient.</p>
<p>This sets the skeleton of this post&rsquo;s answer: turn-level reward enters training <strong>not by weighting environment tokens, but by constructing a separate advantage for each turn&rsquo;s model tokens</strong>.</p>
<h2 id="three-layers-of-signal-how-reward-travels-from-trajectory-to-weights">Three Layers of Signal: How Reward Travels from Trajectory to Weights</h2>
<p>The textbook splits the signal flow into three layers, and this layering is the map for understanding the whole spectrum of approaches:</p>
<ol>
<li><strong>Trajectory reward</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><msub><mi>τ</mi><mi>i</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">R(\tau_i)</annotation></semantics></math></span>: the raw scalar given by the environment—&ldquo;did this episode succeed or not&rdquo;;</li>
<li><strong>Step advantage</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>: breaks the final outcome down into per-turn scores, answering &ldquo;should turn <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span> be rewarded&rdquo;—this is the main battleground of credit assignment;</li>
<li><strong>Token gradient</strong>: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> is multiplied onto the log-probs of that turn&rsquo;s action tokens, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Δ</mi><mi>θ</mi><mo>=</mo><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><msub><mo>∑</mo><mrow><mi>k</mi><mo>∈</mo><msub><mi>a</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow></msub><mi mathvariant="normal">∇</mi><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>k</mi></msub><mo>∣</mo><mo>⋅</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Delta\theta = A_{i,t} \sum_{k \in a_{i,t}} \nabla \log \pi_\theta(y_k \mid \cdot)</annotation></semantics></math></span>.</li>
</ol>
<p>What &ldquo;putting turn-level reward into GRPO&rdquo; actually comes down to is layer 2: GRPO&rsquo;s within-group comparison naturally works at the trajectory level (ranking multiple trajectories for the same problem against each other), so to get down to the turn level you have to define <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> yourself and feed it in.</p>
<h2 id="why-the-naive-approach-fails-a-controlled-experiment">Why the Naive Approach Fails: A Controlled Experiment</h2>
<p>The textbook&rsquo;s Mini Agent Loop experiment gives the most convincing set of numbers. The task: look up Earth&rsquo;s radius → compute the equatorial circumference → verify → answer. Construct a &ldquo;bad&rdquo; trajectory: <strong>only step 2 is wrong</strong> (π taken as 3), steps 1 and 3 are both correct, and the final answer is wrong.</p>
<ul>
<li><strong>ORM (final outcome only)</strong>: on failure, every step&rsquo;s credit is 0—because <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn><mo>×</mo><mi>γ</mi><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">0 \times \gamma = 0</annotation></semantics></math></span>, after the zero signal propagates backward every step gets &ldquo;no signal&rdquo;; if you switch to &ldquo;-1 on failure&rdquo;, the correct search in step 1 receives a penalty of -0.857. Either the innocent get blamed, or everyone simply goes silent.</li>
<li><strong>PRM (each step scored independently)</strong>: correct steps get positive scores and wrong steps negative ones; the discriminability is 19 times that of ORM, and at the same number of steps the task success rate is about 30 percentage points higher. The cost is that every step has to be evaluated—which in real settings means annotation cost or training yet another scoring model.</li>
</ul>
<p>PRM is too expensive and ORM too coarse; the middle-ground routes of 2025–2026 fall into three categories by &ldquo;where the signal comes from&rdquo;, and all of them are essentially looking for a compromise between the two.</p>
<h2 id="the-credit-assignment-spectrum-three-routes-between-orm-and-prm">The Credit-Assignment Spectrum: Three Routes Between ORM and PRM</h2>
<p><strong>Route 1: compare actions from the same state (state-anchored).</strong> GiGPO&rsquo;s idea is that if multiple rollouts within a group converge on some state <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span>, you compare &ldquo;the subsequent returns of the different actions taken from the same situation&rdquo;, and the within-group relative score is the step advantage. Its refined version, HGPO, points out that the same page is not the same context (depending on whether a constraint was missed earlier, the same &ldquo;add to cart&rdquo; can mean something completely different), so it stratifies the comparison groups by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span>-step history and fuses the strata with weights. This route needs no extra annotation at all, but it depends on states being able to collide—web tasks are naturally friendly to it, while in open-ended dialogue collisions almost never happen.</p>
<p><strong>Route 2: redistribute the final reward back to each step (progress redistribution).</strong> SPA-RL trains a progress estimator under the constraint that the per-step contributions sum back to the final reward (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>R</mi><mo>^</mo></mover><mo>=</mo><msub><mo>∑</mo><mi>t</mi></msub><msub><mover accent="true"><mi>c</mi><mo>^</mo></mover><mi>t</mi></msub><mo>≈</mo><mi>R</mi></mrow><annotation encoding="application/x-tex">\hat R = \sum_t \hat c_t \approx R</annotation></semantics></math></span>): the contributions along a successful trajectory sum to nearly 1 and along a failed one to nearly 0, which forces the model to assign positive contribution to the steps that actually push the task forward. IGPO is cheaper: it directly uses &ldquo;the increase in the model&rsquo;s log probability of the ground-truth answer&rdquo; as the turn-level reward—if after turn <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span> the model is more confident in the correct answer (log-prob rises from -4.0 to -1.5), it records <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>r</mi><mrow><mi>I</mi><mi>G</mi></mrow></msup><mo>=</mo><mo>+</mo><mn>2.5</mn></mrow><annotation encoding="application/x-tex">r^{IG} = +2.5</annotation></semantics></math></span>; if the model was led astray, it records a negative value. What it yields is a ready-made turn-level scalar that can go straight into discounted accumulation, with no Monte Carlo value estimation needed; the cost is a dependence on high-quality reference answers, and questions with multiple valid answers get unfairly penalized.</p>
<p><strong>Route 3: change the optimization granularity (the step-aligned paradigm).</strong> StepPO and Turn-PPO take the most radical view: existing algorithms inherit RLHF&rsquo;s token-centric paradigm, but an agent&rsquo;s decisions are naturally turn-grained, so there is a granularity mismatch between token-level optimization and turn-level decisions. Turn-PPO uses PPO instead of GRPO for multi-turn training, on the grounds that trajectory-level within-group advantages have too much variance, and switches to turn-level advantage estimation—each turn has its own advantage, shared among the tokens within that turn. This route is orthogonal to all the methods above: the others change where <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> comes from; this one changes the basic unit of optimization.</p>
<p>Whichever route you take, multi-turn RL has one more problem, along a purely temporal dimension: <strong>the earlier the mistake, the greater the responsibility</strong> (if step 1 goes wrong, every later step unfolds on top of the error). Both the textbook and engineering implementations handle this with backward discounting: working back from the final outcome, the earlier the step, the larger the discount—for a 7-turn interaction in which only the last turn has reward 1.0 and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi><mo>=</mo><mn>0.9</mn></mrow><annotation encoding="application/x-tex">\gamma = 0.9</annotation></semantics></math></span>, the returns assigned to the turns are <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">[</mo><mn>0.531</mn><mo separator="true">,</mo><mn>0.590</mn><mo separator="true">,</mo><mn>0.656</mn><mo separator="true">,</mo><mn>0.729</mn><mo separator="true">,</mo><mn>0.810</mn><mo separator="true">,</mo><mn>0.900</mn><mo separator="true">,</mo><mn>1.000</mn><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">[0.531, 0.590, 0.656, 0.729, 0.810, 0.900, 1.000]</annotation></semantics></math></span>.</p>
<h2 id="where-the-grpogspo-family-sits-on-this-map">Where the GRPO/GSPO Family Sits on This Map</h2>
<p>This post settles the loose end left by the <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">GSPO post</a>
. GSPO&rsquo;s principle that &ldquo;the unit of optimization should match the unit of reward&rdquo; runs into real tension in multi-turn settings: GSPO&rsquo;s objective treats the entire response as a single action, which suits outcome-level rewards; multi-turn RL wants turn-level credit assignment, which amounts to demanding a granularity finer than the sequence. The GSPO-token variant that §4.3 of the GSPO paper designs for exactly this has a very telling formula:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>s</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>⋅</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><mrow><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">s_{i,t}(\theta) = \mathrm{sg}[s_i(\theta)] \cdot \frac{\pi_\theta(y_{i,t}|x,y_{i,&lt;t})}{\mathrm{sg}[\pi_\theta(y_{i,t}|x,y_{i,&lt;t})]}</annotation></semantics></math></span><p><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><mo>⋅</mo><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\mathrm{sg}[\cdot]</annotation></semantics></math></span> is stop-gradient (it takes only the value and passes no gradient back). The beauty of this construction: numerically, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">s_{i,t}</annotation></semantics></math></span> is identically equal to the sequence-level ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>, so the objective, the clipping condition, and the theoretical gradient are fully equivalent to GSPO&rsquo;s; yet the gradient expression now contains a &ldquo;per-token customizable advantage slot&rdquo;. Set all tokens of the same turn to the same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> and you get exactly turn-level credit assignment—this is a formal opening the algorithm layer leaves for multi-turn training, not an engineering hack.</p>
<p>The GRPO family takes the other, symmetric route: it leaves the mechanism of comparing multiple trajectories within a group untouched and outsources the construction of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>—whether you hand it trajectory-level comparisons (the default) or turn-level scores (ORM + discounting / PRM / IGPO / SALT), it multiplies them in all the same. So <strong>in implementation terms, &ldquo;turn-level reward in GRPO&rdquo; is not a change to the algorithm but a change to the advantage constructor + making sure the action mask is clean</strong>. The combination of the two routes at the paper level (GSPO-token × turn-level <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>) currently has no public ablation—a genuine gap.</p>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>the number of turns sets the tier; the algorithm can&rsquo;t.</strong> The textbook&rsquo;s selection guide tiers by turn count: for 3–5 turns, plain ORM/GRPO is enough (episodes are short, so sparsity isn&rsquo;t fatal); for 5–15 turns, bring in milestone rewards or annotation-free methods like SALT/GiGPO; beyond 15 turns, PRM or progress rewards plus tree search are a must. This tiering matters more than any single technique—when reward density can&rsquo;t sustain learning, switching to a stronger RL algorithm yields zero gain. The five-turn trajectory from the top of this post actually sits right on the boundary between the first and second tiers.</p>
<p>Second, <strong>the link between credit-assignment quality and entropy is more direct than you might think.</strong> Under ORM every step of a failed trajectory gets penalized, and the easiest countermeasure the model learns is to reduce behavioral diversity (take fewer actions, answer earlier)—in the multi-turn setting, entropy collapse is not a by-product but a direct consequence of wrong credit assignment. Conversely, fine-grained signals like PRM or IGPO let the model &ldquo;keep the right steps and fix only the wrong ones&rdquo;, which is what preserves exploration. In multi-turn training, watching the agent&rsquo;s turn-count distribution and tool-call diversity exposes credit-assignment problems earlier than watching the reward curve.</p>
<p>Third, <strong>four reward types stand between &ldquo;it runs&rdquo; and &ldquo;the credit is assigned correctly&rdquo;.</strong> The textbook divides agentic rewards into four categories: Outcome (is the final result correct), Format (can the action be parsed and executed), Cost (number of turns, API spend), and Process (does the step advance the task). When getting started, people usually implement only the first two, to make sure training runs, and Process is exactly what the credit-assignment layer needs to make dense. But many engineering setups also fold the Format penalty into the outcome signal and assign credit for both together—a turn with a format error and a turn with a reasoning error bear the same blame in the trajectory, and almost no public report teases this kind of conflation apart.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>There is no public ablation of how GSPO-token combined with turn-level <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> performs on multi-turn agentic tasks—whether Qwen has validated it internally is impossible to know;</li>
<li>The <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi></mrow><annotation encoding="application/x-tex">\gamma</annotation></semantics></math></span> of turn-level discounting has not been systematically swept in multi-turn LLM training (the textbook example uses 0.9, which comes from classical RL convention rather than an ablation);</li>
<li>The mix and isolation of the four reward types (outcome/format/cost/process) at the credit-assignment layer lack public controlled experiments—most reports give only final metrics, not the credit-assignment structure.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
: 22.2 Multi-turn RL formalization (AppWorld POMDP, action mask, the four reward types) and 22.3 Trajectory credit assignment (three signal layers, the ORM/PRM experiment, three middle-ground routes, turn-level discounting, selection tiers), in <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter22_agentic/formulation.md"><code>docs/chapter22_agentic/formulation.md</code></a>
 and <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter22_agentic/credit-assignment.md"><code>docs/chapter22_agentic/credit-assignment.md</code></a>
</li>
<li>GSPO paper: <a href="https://arxiv.org/abs/2507.18071">arXiv:2507.18071</a>
 §4.3 (the GSPO-token formula and the equivalence argument)</li>
<li>AppWorld: <a href="https://arxiv.org/abs/2407.18901">AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents</a>
; original ORM/PRM framework: <a href="https://arxiv.org/abs/2305.20050">Let&rsquo;s Verify Step by Step</a>
 (Lightman et al., 2023)</li>
<li>Middle-ground routes: SALT [2510.20022], GiGPO [2505.10978], HGPO [2602.22817], SPA-RL [2505.20732], IGPO [2510.14967], Turn-PPO [2512.17008], StepPO (the ID given in the textbook points to a different paper; the original could not be verified) (arXiv IDs checked against arXiv)</li>
<li>Previous posts: <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">The Ratio&rsquo;s Wrong Unit: GSPO Goes Sequence-Level</a>
, <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>One Vote or Ten: token-mean vs sample-mean</title>
      <link>https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/</link>
      <pubDate>Sun, 30 Aug 2026 12:07:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/</guid>
      <description>Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.</description>
      <content:encoded><![CDATA[<p>In the same batch, which contributes more to the gradient: a 500-token response or a 5000-token one? GRPO says they count equally, DAPO says the long one counts ten times as much, and both can justify their answer. This is a dispute over loss aggregation: sample-mean gives each response one vote, token-mean gives each token one vote. The DAPO paper (§3.3) says long-CoT settings must switch to token level, but the switch pushes the length skew from the response level to the question level. This post unpacks the three formulas, the two camps&rsquo; arguments, and a tradeoff few people talk about.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/token-mean-vs-sample-mean/">interactive report</a>
 (in Chinese), with a weight calculator that switches among the three vote-counting rules and a side-by-side breakdown of the formulas.</p>
</blockquote>
<h2 id="first-count-the-votes">First, Count the Votes</h2>
<p>The same training batch contains two responses: one is 500 tokens, the other 5000 tokens. How much does each contribute to this step&rsquo;s gradient?</p>
<ul>
<li><strong>GRPO (sample-mean)</strong>: equally. First average over the tokens within each response, then average over the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses — every response, long or short, gets one equal vote;</li>
<li><strong>DAPO (token-mean)</strong>: the long one contributes ten times as much. All tokens in the batch are flattened together and divided by the total token count — one vote per token, so the 5000-token response gets 5000 votes;</li>
<li><strong>Dr.GRPO</strong>: formally also one vote per response, but the divisor changes from &ldquo;each response&rsquo;s actual length&rdquo; to a global constant, which changes what a vote means (that is the topic of the <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">Dr.GRPO post</a>
).</li>
</ul>
<p>All three answers are backed by papers, and all three camps are working on <strong>the same spot in the same expression</strong> — the loss&rsquo;s aggregation divisor. This knob is orthogonal to the advantage normalization (the std half) covered in the Dr.GRPO post: that post&rsquo;s question was &ldquo;how is the advantage computed?&rdquo;; this post&rsquo;s question is &ldquo;once it is computed, by what rule are the tokens&rsquo; losses added together?&rdquo;</p>
<h2 id="three-formulas-side-by-side">Three Formulas Side by Side</h2>
<p>The GRPO objective (DeepSeekMath&rsquo;s Equation 3, with the KL term omitted):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="script">J</mi><mrow><mi>G</mi><mi>R</mi><mi>P</mi><mi>O</mi></mrow></msub><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><msub><mi mathvariant="normal">ℓ</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo fence="true">]</mo></mrow><mo separator="true">,</mo><mspace width="1em"/><msub><mi mathvariant="normal">ℓ</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo>=</mo><mi>min</mi><mo>⁡</mo><mo fence="false" stretchy="true" minsize="1.2em" maxsize="1.2em">(</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo separator="true">,</mo><mtext> </mtext><mrow><mi mathvariant="normal">c</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">p</mi></mrow><mo stretchy="false">(</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo separator="true">,</mo><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo fence="false" stretchy="true" minsize="1.2em" maxsize="1.2em">)</mo></mrow><annotation encoding="application/x-tex">\mathcal{J}_{GRPO} = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\ell_{i,t}\right], \quad \ell_{i,t}=\min\big(w_{i,t}\hat{A}_i,\ \mathrm{clip}(w_{i,t},1-\varepsilon,1+\varepsilon)\hat{A}_i\big)</annotation></semantics></math></span><p>Here <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>o</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">o_i</annotation></semantics></math></span> is the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span>-th response, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">|o_i|</annotation></semantics></math></span> is its length, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">w_{i,t}</annotation></semantics></math></span> is the probability ratio between the new and old policies for that token, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\hat{A}_i</annotation></semantics></math></span> is the advantage shared by the entire response. Note the structure of the sums: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><msub><mo>∑</mo><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\frac{1}{|o_i|}\sum_t</annotation></semantics></math></span> first averages each response <strong>on its own</strong> into a single number, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mi>G</mi></mfrac><msub><mo>∑</mo><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\frac{1}{G}\sum_i</annotation></semantics></math></span> then averages those <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> numbers. The two responses each cast one vote, regardless of length.</p>
<p>DAPO changes just one spot (Equation 12 of the paper, the part highlighted in red):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="script">J</mi><mrow><mi>D</mi><mi>A</mi><mi>P</mi><mi>O</mi></mrow></msub><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mrow><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><msub><mi mathvariant="normal">ℓ</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\mathcal{J}_{DAPO} = \mathbb{E}\left[\frac{1}{\sum_{i=1}^{G}|o_i|}\sum_{i=1}^{G}\sum_{t=1}^{|o_i|}\ell_{i,t}\right]</annotation></semantics></math></span><p>The inner <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac></mrow><annotation encoding="application/x-tex">\frac{1}{|o_i|}</annotation></semantics></math></span> is gone, the two levels of summation are flattened into one, and the divisor becomes the total token count of the whole group. Every token gets exactly the same weight, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><msub><mo>∑</mo><mi>i</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/\sum_i|o_i|</annotation></semantics></math></span>; a response&rsquo;s total weight is proportional to its length.</p>
<p>The third is Dr.GRPO: the same aggregation structure as DAPO (tokens flattened), but the divisor is a global constant (such as the length cap), and the std is removed from the advantage as well — the Dr.GRPO post went into that in detail; this one looks only at the divisor.</p>
<h2 id="why-dapo-insists-on-switching-two-places-the-penalty-cant-reach">Why DAPO Insists on Switching: Two Places the Penalty Can&rsquo;t Reach</h2>
<p>The argument in §3.3 of the DAPO paper is quite concrete and targets the long-CoT setting. Under sample-mean, long samples &ldquo;contribute disproportionately little to the overall loss&rdquo; (original wording: disproportionately lower contribution), which leads to two bad outcomes:</p>
<p>First, <strong>reasoning patterns in high-quality long samples don&rsquo;t get learned</strong>. In a correct 5000-token chain of reasoning, each token receives a gradient weight of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mi>G</mi><mo>⋅</mo><mn>5000</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">1/(G\cdot 5000)</annotation></semantics></math></span>, only one-tenth of what a token in a 500-token short sample gets. What long samples carry is precisely complex reasoning, so under sample-mean, the more important the content, the lower its per-unit weight.</p>
<p>Second, <strong>bad patterns in overlong samples don&rsquo;t get penalized away</strong>. The paper reports an observation: overly long samples often contain low-quality patterns such as gibberish (garbled text) and repeated words. The penalty on these bad tokens is likewise diluted by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|o_i|</annotation></semantics></math></span> — the longer the sample, the more lightly each bad token is penalized. The result is &ldquo;an unhealthy increase in entropy and response length&rdquo; (the measured curves in Figures 4a/4b of the paper).</p>
<p>After switching to token-mean, a token carries the same weight whether it sits in a long sample or a short one. DAPO is upfront about the effect: as a result, longer sequences &ldquo;have more influence on the overall gradient update than shorter ones&rdquo; (original wording: longer sequences can have more influence). This is not a side effect but a design goal — in the long-CoT era, the gradient should be spent mainly on long responses in the first place.</p>
<h2 id="a-tradeoff-few-people-talk-about">A Tradeoff Few People Talk About</h2>
<p>But the phrase &ldquo;long samples get more votes&rdquo; deserves a pause and a second look: it has two layers of meaning, and DAPO only spells out the first.</p>
<p>The first layer is the response level: within a single response, tokens are no longer diluted by length. This is a direction Dr.GRPO endorses too — the &ldquo;response-level length bias&rdquo; that Dr.GRPO criticizes (when correct, shorter responses are favored; when wrong, longer ones are safer) and the &ldquo;bad tokens don&rsquo;t get penalized&rdquo; problem that DAPO criticizes are two faces of the same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|o_i|</annotation></semantics></math></span>, and the two camps&rsquo; prescriptions reach the same destination by different roads: equal weight per token.</p>
<p>The second layer is the question level: across the whole batch, <strong>questions with long responses take more of the total weight</strong>. A question on which the model&rsquo;s responses average 6000 tokens and one on which they average 600 tokens differ in batch weight by 10x under token-mean; under sample-mean the two are equal. In other words, token-mean pushes the length skew from the response level to the question level — long questions dominate the gradient. In the long-CoT setting this may be exactly what you want (hard, reasoning-heavy questions tend to have long responses too), but it is a real bias: question weights are no longer uniform; instead, each question is overweighted in proportion to its current length. I have not seen this layer of the tradeoff quantified in any paper (this point is an inference; happy to be proven wrong).</p>
<p>There is also a second second-order effect: DAPO&rsquo;s divisor <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mo>∑</mo><mi>i</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">\sum_i|o_i|</annotation></semantics></math></span> is <strong>a random variable that varies from batch to batch</strong>, whereas Dr.GRPO&rsquo;s constant divisor is not. The direct consequence of a random divisor is that the gradient scale fluctuates with the length composition of each batch; the indirect consequence is more interesting — later in training, responses generally get longer, the divisor grows, and each token&rsquo;s effective learning rate decays automatically. In other words, token-mean comes with a built-in implicit learning-rate decay that moves in lockstep with length inflation, a property that neither sample-mean nor a constant divisor has. This is a mechanism-level inference that no paper has measured, but it is a real variable when choosing an aggregation method: what you are choosing is not just a weight allocation but also the dynamics of the gradient scale.</p>
<h2 id="both-say-back-to-ppo-but-not-the-same-ppo">Both Say &ldquo;Back to PPO,&rdquo; but Not the Same PPO</h2>
<p>An easily overlooked historical fact: PPO&rsquo;s original objective was aggregated per token to begin with (summing or averaging over the tokens of the whole batch, with GAE supplying a per-token advantage). <strong>It is GRPO&rsquo;s sample-mean that is the outlier</strong> — because GRPO&rsquo;s advantage is &ldquo;one number per response,&rdquo; averaging each response into one number first and then averaging across responses is semantically the tidiest.</p>
<p>Then DAPO and Dr.GRPO both claim to have &ldquo;restored the PPO objective&rdquo;: DAPO restores <strong>PPO&rsquo;s aggregation scheme</strong> (tokens flattened, but the advantage keeps its z-score, so the std is still there); Dr.GRPO restores <strong>the form of PPO&rsquo;s objective function</strong> (a constant divisor, the advantage reduced to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>−</mo><mtext>mean</mtext></mrow><annotation encoding="application/x-tex">r-\text{mean}</annotation></semantics></math></span>, the std removed). The two knobs — aggregation granularity and advantage normalization — are independent and can be combined freely: in implementations such as verl, the two really are configured separately (from the implementation-level discussion in the Dr.GRPO post: by default, <code>masked_mean</code> in trl/verl adds back the per-response averaging that the formula doesn&rsquo;t have, and auditing whether each divisor &ldquo;is a variable or a constant&rdquo; is the cheapest check there is). This also means that &ldquo;GRPO vs DAPO vs Dr.GRPO&rdquo; is not three algorithms lined up side by side, but a configuration space of at least 2×2, and the good points known so far are all still being found by trial and error along its edges.</p>
<h2 id="which-vote-should-weigh-more-and-when">Which Vote Should Weigh More, and When</h2>
<p>Break it down by scenario and the choice isn&rsquo;t mysterious:</p>
<ul>
<li><strong>Short CoT, low length variance</strong>: the two aggregations are approximately equivalent (lengths are similar, so <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|o_i|</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mo>∑</mo><mi mathvariant="normal">∣</mi><msub><mi>o</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/\sum|o_i|</annotation></semantics></math></span> differ by a roughly constant factor), so there&rsquo;s no need to agonize;</li>
<li><strong>Long CoT, bad patterns hiding in long samples</strong> (repetition, gibberish, pointless rambling): token-mean is crucial, otherwise bad tokens will never be penalized enough — this is DAPO&rsquo;s core scenario;</li>
<li><strong>You want uniform weights across questions</strong> (e.g., question difficulty varies widely and you worry about long questions dominating): sample-mean or a constant divisor is more controllable;</li>
<li><strong>You want a stable gradient scale and predictable learning-rate behavior</strong>: a constant divisor (the Dr.GRPO route) is the cleanest; token-mean&rsquo;s random divisor mixes length dynamics into the learning rate.</li>
</ul>
<p>My take: at its core, the aggregation dispute is about &ldquo;in what unit the gradient budget is allocated.&rdquo; Under sample-mean it is allocated per response, implicitly assuming every response carries the same amount of information — true in the era of short answers; under token-mean it is allocated per token, implicitly assuming every token carries the same amount of information — closer to the truth in the long-CoT era, but at the cost of letting length decide the weighting across questions. The paper that is really missing is one that makes this budget allocation <strong>explicit</strong>: weights assigned neither per response nor per token, but according to estimated information content or uncertainty. Until then, DAPO&rsquo;s change is the most pragmatic default for long CoT.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>The effect of token-mean&rsquo;s question-level length skew (cross-question weight ∝ length) on where training ends up has not been publicly quantified;</li>
<li>Nobody has measured how the implicit learning-rate decay from the random divisor interacts with an explicit LR schedule stacked on top of it;</li>
<li>For a full 2×2 ablation of the two knobs, aggregation granularity and advantage normalization (sample/token × with std/without std), the public literature only has the points on the diagonal.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>DAPO paper: <a href="https://arxiv.org/abs/2503.14476">arXiv:2503.14476</a>
; token-level loss in §3.3 (Equation 12), entropy and length curves in Figure 4</li>
<li>Dr.GRPO paper: <a href="https://arxiv.org/abs/2503.20783">arXiv:2503.20783</a>
; the original statements of the response-level length bias and the question-level difficulty bias</li>
<li>DeepSeekMath / GRPO: <a href="https://arxiv.org/abs/2402.03300">arXiv:2402.03300</a>
; the sample-mean objective (Equation 3)</li>
<li>Previous post (covering the advantage-normalization half): <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">Dr.GRPO: Two Normalization Terms, Two Biases</a>
 (it gave the token-mean section number as §3.2 by mistake; the correct one is §3.3, and that has been corrected in passing)</li>
<li>The question-level length skew and the random divisor&rsquo;s implicit LR decay are this post&rsquo;s own inferences, each flagged where it appears in the text</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>Judging Truncated Answers: DAPO&#39;s Overlong Fix</title>
      <link>https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/</link>
      <pubDate>Sun, 30 Aug 2026 12:05:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/</guid>
      <description>Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.</description>
      <content:encoded><![CDATA[<p>RL training needs a generation length cap, and truncated responses are treated as wrong by default—which misjudges &lsquo;undecidable&rsquo; samples as &lsquo;wrong&rsquo; ones and injects reward noise straight into the group-relative advantage. DAPO&rsquo;s two fixes: Overlong Filtering acknowledges that such samples are undecidable and masks their loss; Soft Overlong Punishment builds a linear buffer ramp before the cliff, turning length control into a continuous signal. This post unpacks the mechanism, the formula, and the configuration values, and fits it together with Dr.GRPO and token-mean into a complete picture of the length problem.</p>
<blockquote>
<p>This post comes with a companion <a href="https://njx-njx.github.io/reports/dapo-overlong-reward-shaping/">interactive report</a>
 (in Chinese), featuring a drill-down length ruler, a three-piece breakdown of Equation 13, and a switchable comparison of the three handling approaches.</p>
</blockquote>
<h2 id="an-engineering-setting-you-cant-avoid">An Engineering Setting You Can&rsquo;t Avoid</h2>
<p>Rollouts in RL training must have a generation length cap. GPU memory has to be reserved for the longest sequence, the batch has to wait for the slowest response to finish, and a single runaway response can drag down the throughput of the whole batch. DAPO&rsquo;s configuration: an expected maximum length of 16,384 tokens, an additional 4,096-token buffer, and a hard generation cap of 20,480 (paper §4.1).</p>
<p>Once a cap is set, some responses will inevitably be truncated. What should happen to these samples? The default practice is common: assign a punitive reward and treat them as wrong answers (reward recorded as 0 or a negative score).</p>
<p>The DAPO paper (arXiv:2503.14476) points out in §3.4 that this default causes problems: <strong>improper handling of truncated samples introduces reward noise, and that noise is injected into the entire training process, amplified by within-group normalization</strong>. The paper&rsquo;s reasoning is straightforward: when a reasoning process that is sound in itself gets penalized merely for running too long, the penalty confuses the model—it can&rsquo;t tell whether its reasoning was valid.</p>
<h2 id="how-the-noise-gets-into-the-advantage">How the Noise Gets into the Advantage</h2>
<p>GRPO&rsquo;s advantage is a within-group relative ranking: sample <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses for the same problem and subtract the group mean (dividing by the std). This means that <strong>once the reward of any single response is wrong, it affects the whole group</strong>.</p>
<p>Take a concrete example: 8 responses are sampled for a problem, and the true split is 4 right and 4 wrong; among them, 2 &ldquo;wrong&rdquo; responses are actually high-quality reasoning that got truncated (had they been finished, they would likely have been right).</p>
<ul>
<li>By true quality: the group mean is 0.5, the correct responses get positive advantages, and the wrong ones get negative ones;</li>
<li>Scoring truncation as 0: the truncated samples are also counted as wrong answers and the group mean is dragged down (6 counted as wrong, mean 0.25), so the advantage of every correct sample is inflated and the negative advantage of every wrong sample is softened—including the two that genuinely got it wrong.</li>
</ul>
<p>As if one layer of distortion weren&rsquo;t enough, there is a second: those two truncated high-quality reasoning traces themselves get reinforced as negative samples. What the model learns from them is not &ldquo;write a bit shorter&rdquo; but &ldquo;my way of reasoning is wrong&rdquo;—which is exactly where the paper says the model gets confused. Once these two kinds of penalty are lumped together, the model throws out the good way of reasoning along with &ldquo;writing long&rdquo;.</p>
<h2 id="approach-1-overlong-filteringdeem-truncated-samples-undecidable-and-mask-their-loss">Approach 1: Overlong Filtering—Deem Truncated Samples &ldquo;Undecidable&rdquo; and Mask Their Loss</h2>
<p>DAPO&rsquo;s first approach: rather than treating a truncated sample as a wrong sample, simply refuse to recognize it as a sample at all. Overlong Filtering masks the entire loss of a truncated response—the sample takes no part in advantage estimation and produces no gradient.</p>
<p>The effect shown in Figure 5 of the paper: after applying Overlong Reward Shaping, both AIME accuracy and the actor model&rsquo;s entropy become more stable, and training stability improves markedly. This is worth noting: the stabilized entropy means truncation noise itself is interfering with the exploration dynamics—it doesn&rsquo;t merely affect the reward numbers on paper.</p>
<p>The costs are just as obvious. The paper doesn&rsquo;t go into them, so I&rsquo;ll add two points:</p>
<ol>
<li><strong>Wasted compute</strong>. The rollout cost of a truncated sample has already been paid (it is often the longest response, which happens to consume the most compute), and once it is masked, not a single gradient comes back in return. This is the same logic as dynamic sampling—both &ldquo;trade compute for signal quality&rdquo;, which is the orientation of the whole DAPO paper;</li>
<li><strong>Hidden survivorship bias</strong>. Which samples are more likely to be truncated? Long problems, hard problems, problems that need long CoT. Systematically masking truncated samples amounts to systematically down-weighting hard problems in the training distribution. In the short term, training becomes more stable; the long-term cost is that the model may reveal weaknesses on exactly the problems that most need long reasoning. The paper doesn&rsquo;t discuss this side effect; this reservation is my own.</li>
</ol>
<h2 id="approach-2-soft-overlong-punishmentbuild-a-buffer-ramp-before-the-cliff">Approach 2: Soft Overlong Punishment—Build a Buffer Ramp Before the Cliff</h2>
<p>Masking everything loses information: the model can&rsquo;t learn that length itself matters. DAPO&rsquo;s second approach is Soft Overlong Punishment, which turns length itself into a <strong>continuous, discriminable reward dimension</strong> (Equation 13 in the paper):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>R</mi><mrow><mi>l</mi><mi>e</mi><mi>n</mi><mi>g</mi><mi>t</mi><mi>h</mi></mrow></msub><mo stretchy="false">(</mo><mi>y</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mo fence="true">{</mo><mtable rowspacing="0.36em" columnalign="left left" columnspacing="1em"><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mn>0</mn><mo separator="true">,</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi><mo>≤</mo><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>−</mo><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mstyle displaystyle="true" scriptlevel="0"><mfrac><mrow><mo stretchy="false">(</mo><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>−</mo><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub><mo stretchy="false">)</mo><mo>−</mo><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi></mrow><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub></mfrac></mstyle><mo separator="true">,</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>−</mo><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub><mo>&lt;</mo><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi><mo>≤</mo><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub></mrow></mstyle></mtd></mtr><mtr><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><mo>−</mo><mn>1</mn><mo separator="true">,</mo></mrow></mstyle></mtd><mtd><mstyle scriptlevel="0" displaystyle="false"><mrow><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>&lt;</mo><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi></mrow></mstyle></mtd></mtr></mtable></mrow></mrow><annotation encoding="application/x-tex">R_{length}(y) = \begin{cases} 0, &amp; |y| \le L_{max} - L_{cache} \\ \dfrac{(L_{max} - L_{cache}) - |y|}{L_{cache}}, &amp; L_{max}-L_{cache} &lt; |y| \le L_{max} \\ -1, &amp; L_{max} &lt; |y| \end{cases}</annotation></semantics></math></span><p>The three pieces, one at a time:</p>
<ul>
<li><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi><mo>≤</mo><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub><mo>−</mo><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub></mrow><annotation encoding="application/x-tex">|y| \le L_{max}-L_{cache}</annotation></semantics></math></span> (i.e., ≤ 12,288 tokens under DAPO&rsquo;s configuration): no penalty. Length is completely free in this regime;</li>
<li>Entering the cache interval (12,288 to 16,384): a linearly ramping penalty, falling smoothly from 0 to -1. Every extra token written increases the penalty by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><msub><mi>L</mi><mrow><mi>c</mi><mi>a</mi><mi>c</mi><mi>h</mi><mi>e</mi></mrow></msub><mo>=</mo><mn>1</mn><mi mathvariant="normal">/</mi><mn>4096</mn></mrow><annotation encoding="application/x-tex">1/L_{cache} = 1/4096</annotation></semantics></math></span>;</li>
<li>Beyond <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub></mrow><annotation encoding="application/x-tex">L_{max}</annotation></semantics></math></span> (truncated): the penalty is capped at -1. Note that this term is <strong>added to the original correctness reward</strong>—a response that is correct but runs over can still have its total score pulled down.</li>
</ul>
<p>The most ingenious thing about this design: <strong>the cliff is not removed; a buffer ramp is built in front of it</strong>. Going past <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>L</mi><mrow><mi>m</mi><mi>a</mi><mi>x</mi></mrow></msub></mrow><annotation encoding="application/x-tex">L_{max}</annotation></semantics></math></span> is still a hard -1 penalty, just as harsh as the naive approach; but the model starts receiving a continuous &ldquo;almost over budget&rdquo; signal four thousand tokens before the cliff, and the signal strength is monotonically related to how far over budget it is. After within-group normalization, two responses—&ldquo;12,500 tokens, nearly over&rdquo; and &ldquo;16,300 tokens, about to go over&rdquo;—get a distinguishable advantage gap, from which the model can learn to <strong>wrap up early</strong> instead of only feeling the pain once it hits the cap. Length control turns from a binary event (not over / over) into a continuous, learnable task.</p>
<h2 id="placing-it-in-the-full-length-problem-picture">Placing It in the Full &ldquo;Length Problem&rdquo; Picture</h2>
<p>The previous few posts in this series all had to do with length; this post covers the final aspect:</p>
<ul>
<li><strong>Gradient dilution from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y|</annotation></semantics></math></span></strong> (Dr.GRPO, see the <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">normalization-bias post</a>
): the longer a wrong response, the smaller the penalty on each token, so the model tends to make wrong responses overly long. Fix: change the normalization term;</li>
<li><strong>token-mean</strong> (DAPO §3.3): instead of averaging within each sample first and then summing over the group, average uniformly over all tokens in the batch, which keeps long responses from taking an outsized weight in the loss. Fix: change the loss aggregation;</li>
<li><strong>Overlong</strong> (DAPO §3.4, this post): the truncated samples produced by the generation length cap have undecidable rewards. Fix: mask them (filtering) or change the reward function (soft punishment).</li>
</ul>
<p>The three address length problems at three different levels: gradient weighting, loss aggregation, and reward judgment. DAPO uses both token-mean and overlong shaping, meaning it fixes both the gradient side and the reward side; Dr.GRPO focuses on normalization. They don&rsquo;t conflict.</p>
<p>One can also see a design orientation in DAPO: <strong>better to use no signal than an unreliable one</strong>. Dynamic sampling filters out prompts with no gradient signal (all-correct / all-wrong problems), overlong filtering masks samples with unreliable rewards (truncated responses), and Dataset Transformation converts all answers to integers to eliminate parser noise (§3.5, DAPO-Math-17K). All four techniques point in the same direction: training collapse is often not because the algorithm isn&rsquo;t good enough, but because noise has crept into the signal.</p>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>truncated samples are &ldquo;undecidable&rdquo; samples, and judging them as wrong is a misjudgment at the reward level</strong>. The premise of RLVR is that rewards are verifiable—you assign a score only if you can judge right or wrong. For a truncated sample, the verifier never got to finish its judgment at all; it is neither a 1 nor a 0. Overlong Filtering acknowledges this undecidability, while Soft Punishment introduces a new decidable dimension (length itself). Both approaches are consistent with the principle that &ldquo;rewards must be verifiable&rdquo;, whereas the naive punitive reward violates it.</p>
<p>Second, <strong>the fact that entropy is more stable after overlong handling matters more than it appears</strong>. Figure 5(b) shows that the actor&rsquo;s entropy curve is smoother after shaping. This indicates that the chaos near the length cap (high-quality long reasoning that suddenly gets penalized, responses forced into hasty endings) doesn&rsquo;t merely distort the reward; it also adds exploration noise. Read this together with the <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Clip-Higher post</a>
 (entropy collapse): entropy can collapse when wrongly suppressed by the optimizer, and it can also swing wildly when disturbed by reward noise. Entropy monitoring has to be analyzed together with the quality of reward judgments.</p>
<p>Third, <strong>the cache width and the masking ratio are two hyperparameters that need monitoring, and there is currently no public basis for them</strong>. The paper doesn&rsquo;t say how the buffer-ramp length of 4,096/16,384 = 25% was chosen; during training, the truncation rate (the fraction of masked samples) drifts with the model&rsquo;s length behavior—when the masking rate is too high, the effective batch shrinks and the share of hard problems drops; when it is too low, that indicates length control has already broken down or the cap is set too loose. The DAPO paper has no monitoring curves for either number, so reproductions have to fill them in themselves.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>There are no public experiments on whether the hard-problem weight lost to Overlong Filtering needs to be compensated with a progressive length curriculum in long training runs;</li>
<li>The rationale and ablations for the choice of cache width (25%) are not disclosed in the paper;</li>
<li>There is currently no data on whether long but correct exploratory responses get doubly suppressed when Soft Punishment&rsquo;s -1 cap is stacked with Clip-Higher&rsquo;s asymmetric clipping.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>DAPO paper: <a href="https://arxiv.org/abs/2503.14476">arXiv:2503.14476v2</a>
 (source of the §3.4 mechanism description, Equation 13, the §4.1 configuration values, and Figure 5)</li>
<li>Textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
: Chapter 18 <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter18_grpo/deepseek-dapo.md"><code>docs/chapter18_grpo/deepseek-dapo.md</code></a>
 and the appendix DAPO cheatsheet (<a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/en/appendix_code_cheatsheet/dapo.md"><code>docs/en/appendix_code_cheatsheet/dapo.md</code></a>
)</li>
<li>Previous posts: <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
, <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">&ldquo;When Wrong, Write Longer&rdquo;: How GRPO&rsquo;s Two Normalization Terms Distort the Optimization Direction</a>
</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>“Wrong? Write Longer”: What GRPO Normalizers Skew</title>
      <link>https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/</link>
      <pubDate>Sat, 29 Aug 2026 12:05:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/</guid>
      <description>Part of R1-Zero&amp;rsquo;s growing length may be the optimizer&amp;rsquo;s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.</description>
      <content:encoded><![CDATA[<p>R1-Zero&rsquo;s “length and reward rising together” has been treated as the signature curve of emergent reasoning, but Sea AI Lab&rsquo;s Dr.GRPO paper points out that half of that rise may be the optimizer&rsquo;s own preference: 1/|y| makes the model lean toward writing longer when it answers wrong, and dividing by std lets questions at the difficulty extremes take more of the weight. A mechanistic breakdown of two normalization terms and two biases, plus an implementation-level bias hidden in masked_mean.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/dr-grpo-length-and-difficulty-bias/">interactive report</a>
 (in Chinese), with drill-down worked examples of the biases, a side-by-side comparison of masked_mean implementations, and a decision matrix for choosing a fix.</p>
</blockquote>
<h2 id="an-over-trusted-curve">An Over-Trusted Curve</h2>
<p>The most famous plot from R1-Zero training is the one where response length and training reward rise in lockstep. The community was quick to spin a story around it: the model has learned to think longer, and long CoT keeps taking shape through RL.</p>
<p>Sea AI Lab&rsquo;s Dr.GRPO paper (arXiv:2503.20783) poses a simpler question: on the plateau where reward has stopped rising, why does length keep growing? After pinning down the causes one by one, the paper concludes that this is not the model learning to reason more; rather, <strong>the optimizer keeps pushing the model to generate longer responses</strong>. The driving force comes from two normalization terms in the GRPO objective: they adjust the model&rsquo;s optimization direction in opposite ways for correct and incorrect responses, and their effect on incorrect responses points to one and the same outcome: <strong>when the answer is wrong, the longer the response, the less it gets penalized.</strong></p>
<h2 id="bias-1-1y--per-token-averaging-dilutes-the-penalty">Bias 1: 1/|y| — Per-Token Averaging Dilutes the Penalty</h2>
<p>The GRPO objective (Equation 3 in the paper) contains an inconspicuous denominator:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="script">J</mi><mrow><mi>G</mi><mi>R</mi><mi>P</mi><mi>O</mi></mrow></msub><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><mi>min</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo separator="true">,</mo><mtext> </mtext><mrow><mi mathvariant="normal">c</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">p</mi></mrow><mo stretchy="false">(</mo><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo separator="true">,</mo><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo fence="true">)</mo></mrow><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\mathcal{J}_{GRPO} = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^{G} \frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\min\left(w_{i,t}\hat{A}_i,\ \mathrm{clip}(w_{i,t},1-\varepsilon,1+\varepsilon)\hat{A}_i\right)\right]</annotation></semantics></math></span><p>What <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> means is this: within a single response, first average the gradient terms over all of its tokens, then add up the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses. In other words, <strong>the gradient weight each token actually receives is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">\hat{A}_i/|y_i|</annotation></semantics></math></span></strong>. A concrete worked example: sample two responses to the same question, one 100 tokens long and one 1000 tokens long, both with an advantage of absolute value 1 (opposite in sign):</p>
<ul>
<li>Correct (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>=</mo><mo>+</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\hat{A}=+1</annotation></semantics></math></span>): each token of the short response is reinforced by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mn>100</mn><mo>=</mo><mn>0.01</mn></mrow><annotation encoding="application/x-tex">1/100 = 0.01</annotation></semantics></math></span>, while each token of the long response is reinforced by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mn>1000</mn><mo>=</mo><mn>0.001</mn></mrow><annotation encoding="application/x-tex">1/1000 = 0.001</annotation></semantics></math></span>. <strong>Among correct responses, the shorter, the more favored</strong>;</li>
<li>Incorrect (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>=</mo><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">\hat{A}=-1</annotation></semantics></math></span>): each token of the short response is penalized by 0.01, while each token of the long response is penalized by only 0.001. <strong>Among incorrect responses, the longer, the less penalized</strong> — a tenfold difference.</li>
</ul>
<p>This is not an anomaly caused by a math error; it is something the objective writes into every update step: correct answers must be concise; wrong answers may as well be long-winded. Late in training, once reward plateaus, the model&rsquo;s room for improvement on the first half is used up, while the second half keeps pushing the model to generate longer responses — incorrect responses in particular keep getting longer. Fig.5 Plot 2 in the paper&rsquo;s Figure 4 clearly shows this phase: GRPO&rsquo;s length curve keeps rising after reward plateaus.</p>
<h2 id="bias-2-dividing-by-std--extreme-difficulty-questions-take-more-weight">Bias 2: Dividing by std — Extreme-Difficulty Questions Take More Weight</h2>
<p>The second normalization term sits in the advantage:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>=</mo><mfrac><mrow><msub><mi>R</mi><mi>i</mi></msub><mo>−</mo><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mo stretchy="false">(</mo><mo stretchy="false">{</mo><msub><mi>R</mi><mi>j</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy="false">)</mo></mrow><mrow><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">t</mi><mi mathvariant="normal">d</mi></mrow><mo stretchy="false">(</mo><mo stretchy="false">{</mo><msub><mi>R</mi><mi>j</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">\hat{A}_i = \frac{R_i - \mathrm{mean}(\{R_j\}_{j=1}^G)}{\mathrm{std}(\{R_j\}_{j=1}^G)}</annotation></semantics></math></span><p>Dividing by the within-group reward standard deviation is meant to put advantages on a common scale (a standard RL trick). But the paper points out a key difference: classic RL advantage normalization computes the std over <strong>the whole batch</strong>, with all samples sharing one divisor; GRPO <strong>computes the std separately for each question</strong>, so every question has its own divisor — and different divisors mean different weights.</p>
<p>One more worked example: for question 1, all 8 sampled responses are correct (std≈0.35, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span>≈+2.9 on the z-score scale); for question 2, the 8 samples are four correct and four wrong (std≈0.52, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span>≈±1.9). The more extreme the question — the all-correct giveaway, the all-wrong dead end — the smaller its std, and the larger its weight after the division. The model spends the most update budget on questions it already gets entirely right (the easy ones) and on questions it cannot solve at all (never answered correctly even once), while questions that are “partly right, partly wrong, right in the zone where learning pays off” get the least weight. This is what the paper calls <strong>difficulty bias</strong>.</p>
<p>A caveat on scope: std normalization is not all downside. Its design intent, amplifying the signal in low-variance cases, is to let the model learn something even from questions it gets entirely right. What the paper criticizes is the unequal weighting caused by <strong>per-question normalization</strong>, not normalization itself.</p>
<h2 id="bias-3-hidden-in-the-implementation-not-the-formula">Bias 3: Hidden in the Implementation, Not the Formula</h2>
<p>The third finding may hit engineers harder. The PPO objective (Equation 2 in the paper) <strong>sums</strong> over the response&rsquo;s tokens, with no step that divides by length. But the paper examined mainstream open-source implementations and found that the losses in frameworks such as trl and verl all contain a <code>masked_mean</code> line: it averages using the response&rsquo;s actual token count as the divisor (<code>loss.sum(dim) / mask.sum(dim)</code>). In other words, <strong>the formula has no length bias, but the implementation adds one</strong>.</p>
<p>How did this happen? The paper&rsquo;s conjecture traces it to historical practice: in pretraining, all data is packed into fixed-length contexts, and <code>loss.mean(-1)</code>, averaging over the context length, is both numerically convenient and stable; it is division by a constant and introduces no bias. Once this idiom was carried over to the RL stage, the divisor quietly changed from “the fixed context length” to “the actual length of each response” — from a constant to a variable, and the effect changed with it. Dr.GRPO&rsquo;s implementation fix changes just one line: replace the divisor in <code>masked_mean</code> with a global constant (such as the generation length cap MAX_TOKENS).</p>
<p>The formula&rsquo;s authors and the framework&rsquo;s authors may be running the same set of tests, yet they end up with two different gradients. This kind of “formula–implementation mismatch” is more insidious than paper-level biases: you could read the paper ten times and never catch it; only reading the code reveals it.</p>
<h2 id="drgrpos-fix-and-its-justification">Dr.GRPO&rsquo;s Fix and Its Justification</h2>
<p>The fix is simple: remove both normalizations, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> and std, so that the advantage only subtracts the group mean:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>~</mo></mover><mi>i</mi></msub><mo>=</mo><msub><mi>R</mi><mi>i</mi></msub><mo>−</mo><mrow><mi mathvariant="normal">m</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">a</mi><mi mathvariant="normal">n</mi></mrow><mo stretchy="false">(</mo><mo stretchy="false">{</mo><msub><mi>R</mi><mi>j</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\tilde{A}_i = R_i - \mathrm{mean}(\{R_j\}_{j=1}^G)</annotation></semantics></math></span><p>Appendix A of the paper gives three theoretical justifications: with both terms removed, the objective <strong>reverts to the PPO objective (Equation 2)</strong>, and the advantage is just the baseline-free Monte Carlo return plus an unbiased baseline; <strong>subtracting the mean introduces no bias</strong> — the baseline&rsquo;s expected contribution to the gradient is identically zero, a standard result in policy gradients; and the corrected advantage differs from RLOO (Leave-One-Out) only by a constant factor <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mi>G</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">G/(G-1)</annotation></semantics></math></span>, which can be folded into the learning rate and does not affect training dynamics.</p>
<p>The controlled experiments are run in the Oat framework on Qwen2.5-1.5B. The results fall into three layers:</p>
<ul>
<li><strong>Training dynamics</strong>: early on, both GRPO and Dr.GRPO show the R1-Zero-style joint rise of reward and length, but after reward plateaus GRPO&rsquo;s length keeps climbing, while Dr.GRPO&rsquo;s length stabilizes (Fig.5 Plot 1&amp;2);</li>
<li><strong>Behavior</strong>: on benchmarks, the length of Dr.GRPO&rsquo;s incorrect responses drops significantly — the change also mitigates overthinking (Fig.5 Plot 4);</li>
<li><strong>Reproducibility</strong>: after math pretraining, Llama-3.2-3B likewise shows GRPO&rsquo;s length rising together with reward, and Dr.GRPO likewise curbs the continued growth in length (Fig.7). The same recipe also lets the 7B Qwen2.5-Math-7B, using MATH levels 3–5 and training for 27 hours on 8 A100s, reach 43.3% on AIME 2024 — the 7B SOTA at the time, and the whole recipe is extremely simple.</li>
</ul>
<h2 id="a-textbook-erratum-and-the-division-of-labor-with-the-previous-posts">A Textbook Erratum, and the Division of Labor with the Previous Posts</h2>
<p>Section 16.4 of the textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
 also covers this pair of biases, but its wording deserves a correction: it labels “the advantage gets amplified as std approaches zero” as “length bias”, and attributes length inflation to “dividing by std encourages the model to increase within-group variance”. Checked against the original paper, this account is off on two counts: std causes <strong>difficulty bias</strong> (unequal weights across questions), not length bias; and the direct source of length bias is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span>, which has nothing to do with std and acts on the weight of each token within a response. The two normalizations and the two biases each exist independently — the value of the erratum is that choosing a fix depends on this distinction: removing std does not solve the length problem.</p>
<p>The recent posts in this series happen to tackle this at three different levels:</p>
<ul>
<li><strong>DAPO&rsquo;s token-mean</strong> (§3.3): it changes “average within each sample first, then average over the G samples” to “average all tokens in the batch uniformly by the total token count”, i.e. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><msub><mo>∑</mo><mi>i</mi></msub><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><msub><mo>∑</mo><mi>i</mi></msub><msub><mo>∑</mo><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\frac{1}{\sum_i|y_i|}\sum_i\sum_t</annotation></semantics></math></span>. All tokens are thus weighted by the total token count, which formally eliminates length bias — but DAPO&rsquo;s advantage keeps the z-score (including std), so it <strong>only solves half the problem</strong>;</li>
<li><strong>GSPO</strong>: it uses a single ratio at the sequence level and no longer sums over tokens term by term in the objective, so there is no “divide each token by |y|” averaging step at all — it sidesteps this bias structurally;</li>
<li><strong>Dr.GRPO</strong> (this post): it leaves the structure of the objective unchanged and only removes the two variable divisors from the normalization terms.</li>
</ul>
<h2 id="my-three-takeaways">My Three Takeaways</h2>
<p>First, <strong>an RL training curve must be screened for optimizer-induced false signals before anything else</strong>. When “length keeps rising” is cited by academia as headline evidence that “RL is working” and, at the same time, by engineering teams as an anomaly metric that “calls for intervention”, tracing it back to the root matters all the more. The paper also observes that for Qwen2.5-Math models used with a template, the model&rsquo;s pre-RL ability is first damaged by the template and then rebuilt by RL (§3.3) — another class of factor that needs ruling out: part of the “RL gain” is really just ability that was there all along. Before analyzing an RL curve, rule out, one by one, the biases the optimizer introduces and the artifacts the evaluation setup creates.</p>
<p>Second, <strong>Dr.GRPO and dynamic sampling solve different problems; neither replaces the other</strong>. Removing std normalization corrects weight <strong>distortion</strong> (samples that should not be amplified receive too much weight); DAPO&rsquo;s dynamic sampling (questions with std=0 are simply dropped and resampled) deals with <strong>the absence of gradient signal</strong> (when the responses are all correct/all wrong/all zero, the advantage is still 0 after subtracting the mean, and nothing is learned). Note that after removing std, a zero-signal question is still zero-signal — Dr.GRPO brings no gradient to such questions. One ensures correct relative weights within the samples, the other ensures the samples as a whole carry a learnable signal; a real training system needs both configured at once.</p>
<p>Third, <strong>when checking a normalization, first check “whether the divisor is a variable or a constant”</strong>. The three biases here share one pattern: a normalization that looks like routine practice turns out, on closer inspection, to use a denominator that is not a constant — for <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> the divisor is the length of each response, for std it is each question&rsquo;s within-group spread, and for <code>masked_mean</code> it is the mask sum of the current batch. Going through every mean/divide in the training code and asking “can this divisor change from sample to sample?” is the cheapest taste audit there is.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>How the reweighting of “extreme-difficulty questions” after Dr.GRPO removes std plays out in very long training runs (tens of thousands of steps): the time window of the paper&rsquo;s 1.5B/7B experiments is relatively short;</li>
<li>The effective-learning-rate conversion when token-mean (DAPO) and removing the variable divisors (Dr.GRPO) are stacked in the same training stack: there is no public, unified convention;</li>
<li>A systematic check of “how much of the <code>loss.mean(-1)</code> habit from pretraining-era packing still lingers in RL frameworks”: so far there is only the paper&rsquo;s spot check of trl/verl.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Sea AI Lab / NUS: <a href="https://arxiv.org/abs/2503.20783">Understanding R1-Zero-Like Training: A Critical Perspective</a>
 (the source of this post&rsquo;s formulas, experiments, and bias definitions; v2)</li>
<li>Textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
: Section 16.4 (using the corrected wording) and the appendix DAPO cheatsheet (<a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/en/appendix_code_cheatsheet/dapo.md"><code>docs/en/appendix_code_cheatsheet/dapo.md</code></a>
, a side-by-side of DAPO&rsquo;s z-score advantage and the token-mean loss granularity)</li>
<li>RLOO reference: <a href="https://arxiv.org/abs/2402.14740">Back to Basics: Revisiting REINFORCE Style Optimization</a>
</li>
<li>Previous posts: <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
, <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">The Ratio&rsquo;s Wrong Unit: GSPO Goes Sequence-Level</a>
, <a href="https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/">One Model, Two Probabilities: Cost of Mismatch</a>
</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>Samples per Prompt: What GRPO&#39;s Group Size Buys</title>
      <link>https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/</link>
      <pubDate>Sat, 29 Aug 2026 12:00:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/</guid>
      <description>64, 8 or 2 samples per prompt? G is GRPO&amp;rsquo;s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.</description>
      <content:encoded><![CDATA[<p>DeepSeekMath samples 64 responses per question, a teaching implementation defaults to 8, and the latest paper claims even 2 is enough — a 32x spread on the same knob. G is GRPO&rsquo;s only source of a baseline and its biggest compute lever, so what does it actually buy? This post unpacks G&rsquo;s three roles: the sample size of the control variate, the coverage of groups with effective gradients, and the quality of the std-normalization denominator, plus why dynamic sampling can replace a larger G with filtering.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/grpo-group-size-tradeoffs/">interactive report</a>
 (in Chinese), with a zero-gradient-group probability calculator, a dual-curve baseline chart, and a side-by-side comparison of the three camps&rsquo; choices.</p>
</blockquote>
<h2 id="a-knob-with-a-32x-spread">A Knob With a 32x Spread</h2>
<p>On every training step, GRPO samples a group of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses for each prompt. How large this <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> should be is a question on which different camps give wildly different answers:</p>
<ul>
<li>DeepSeekMath (the original GRPO paper, arXiv:2402.03300 §4.2): <strong>64</strong>. With a batch size of 1024, that means a single step covers only 16 prompts;</li>
<li>The RLHF Book&rsquo;s companion teaching code (<code>code/policy_gradients/configs/</code>): <strong>8</strong>, the default across the whole lineup, from GRPO to GSPO and CISPO;</li>
<li>The October 2025 paper &ldquo;It Takes Two: Your GRPO Is Secretly DPO&rdquo; (arXiv:2510.00977): <strong>2</strong> is enough, and the paper argues specifically that the &ldquo;large groups are necessary&rdquo; view is a misreading of why GRPO succeeds.</li>
</ul>
<p>A 32x gap. Worse, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is two things at once: it is GRPO&rsquo;s only source of a baseline (there is no critic; the group mean is the baseline), and it is the biggest multiplier on training cost (rollouts are the main compute expense in RL training, and doubling <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> doubles that cost). One knob governs both signal quality and the compute bill, yet people have gotten everything from 2 to 64 to work — which suggests that the question &ldquo;what does <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> actually buy?&rdquo; is itself worth taking apart.</p>
<h2 id="role-one-the-baselines-sample-size">Role One: The Baseline&rsquo;s Sample Size</h2>
<p>Having dropped the critic, GRPO uses the group mean as its baseline: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>=</mo><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mtext>mean</mtext><mo stretchy="false">(</mo><mo stretchy="false">{</mo><msub><mi>r</mi><mi>j</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\hat{A}_i = r_i - \text{mean}(\{r_j\}_{j=1}^G)</annotation></semantics></math></span> (setting aside the division by std for now). Subtracting a baseline is the standard variance-reduction technique for policy gradients, and the validity condition the textbook (RLHF Book, Chapter 6) gives is that the baseline must not depend on the current action, because <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="double-struck">E</mi><mrow><mi>a</mi><mo>∼</mo><mi>π</mi></mrow></msub><mo stretchy="false">[</mo><mi>b</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>a</mi><mi mathvariant="normal">∣</mi><mi>s</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\mathbb{E}_{a\sim\pi}[b(s)\nabla_\theta\log\pi_\theta(a|s)] = 0</annotation></semantics></math></span> holds for any <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>b</mi><mo stretchy="false">(</mo><mi>s</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">b(s)</annotation></semantics></math></span> that depends only on the state — the baseline leaves the expected gradient unchanged and only suppresses variance.</p>
<p>There is a detail here that few people mention: <strong>GRPO&rsquo;s mean includes the very response being evaluated</strong>, so strictly speaking the baseline depends on the action, and the unbiasedness condition is not strictly satisfied. Compare RLOO (REINFORCE Leave-One-Out, same chapter of rlhf-book): its baseline is the mean of the <strong>other</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>K</mi><mo>−</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">K-1</annotation></semantics></math></span> responses, deliberately excluding the response itself, to &ldquo;keep the gradient estimator fully unbiased.&rdquo; The relationship between the two advantages can be written out directly:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msubsup><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi><mtext>RLOO</mtext></msubsup><mo>=</mo><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mfrac><mn>1</mn><mrow><mi>G</mi><mo>−</mo><mn>1</mn></mrow></mfrac><munder><mo>∑</mo><mrow><mi>j</mi><mo mathvariant="normal">≠</mo><mi>i</mi></mrow></munder><msub><mi>r</mi><mi>j</mi></msub><mo>=</mo><mfrac><mi>G</mi><mrow><mi>G</mi><mo>−</mo><mn>1</mn></mrow></mfrac><mrow><mo fence="true">(</mo><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munder><mo>∑</mo><mi>j</mi></munder><msub><mi>r</mi><mi>j</mi></msub><mo fence="true">)</mo></mrow><mo>=</mo><mfrac><mi>G</mi><mrow><mi>G</mi><mo>−</mo><mn>1</mn></mrow></mfrac><msubsup><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi><mtext>GRPO(no std)</mtext></msubsup></mrow><annotation encoding="application/x-tex">\hat{A}_i^{\text{RLOO}} = r_i - \frac{1}{G-1}\sum_{j\neq i}r_j = \frac{G}{G-1}\left(r_i - \frac{1}{G}\sum_{j}r_j\right) = \frac{G}{G-1}\hat{A}_i^{\text{GRPO(no std)}}</annotation></semantics></math></span><p>The directions are identical; the two differ only by a scale factor of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mi>G</mi><mo>−</mo><mn>1</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">G/(G-1)</annotation></semantics></math></span>. At <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span> the factor is 1.016, which hardly matters; at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">G=8</annotation></semantics></math></span> it is 1.14; <strong>at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">G=2</annotation></semantics></math></span> it is 2</strong> — the GRPO-form advantage is only half as large as the leave-one-out form. This bias is not directional, just a rescaling that grows as <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> shrinks, and in practice its effect can be absorbed by the learning rate (this is an inference, not an experimental finding from any paper), but it explains why every property of small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> gets amplified.</p>
<p>The second entry in the ledger is the baseline&rsquo;s estimation noise. The group mean is an estimate of the true expectation <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="double-struck">E</mi><mo stretchy="false">[</mo><mi>r</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\mathbb{E}[r]</annotation></semantics></math></span> from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> samples, with noise of magnitude roughly <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mi>r</mi></msub><mi mathvariant="normal">/</mi><msqrt><mi>G</mi></msqrt></mrow><annotation encoding="application/x-tex">\sigma_r/\sqrt{G}</annotation></semantics></math></span> (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mi>r</mi></msub></mrow><annotation encoding="application/x-tex">\sigma_r</annotation></semantics></math></span> is the standard deviation of a single response&rsquo;s reward). When <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is small, this noise leaks straight into every response&rsquo;s advantage: a correct response can end up with a negative advantage because &ldquo;everyone else in its group happened to be strong too.&rdquo; The probability that baseline noise randomly flips the gradient direction rises as <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> decreases — this is the standard &ldquo;large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> reduces variance&rdquo; argument, and also what 2510.00977 sets out to challenge; more on that later.</p>
<h2 id="role-two-coverage-of-groups-with-effective-gradients">Role Two: Coverage of Groups With Effective Gradients</h2>
<p>Rewards in RLVR are binary (right or wrong; DAPO uses +1/−1, DeepSeekMath uses 1/0, and the two are equivalent). If a group&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> responses are all right or all wrong, there is no reward variance within the group and every advantage is zero — those <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> rollouts were run for nothing and contribute no gradient at all.</p>
<p>The probability that a group is uniform can be computed directly (what follows is my own calculation): let the model&rsquo;s per-sample accuracy on the question be <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span>, with samples independent; then</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>P</mi><mo stretchy="false">(</mo><mtext>zero-gradient group</mtext><mo stretchy="false">)</mo><mo>=</mo><msup><mi>p</mi><mi>G</mi></msup><mo>+</mo><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><msup><mo stretchy="false">)</mo><mi>G</mi></msup></mrow><annotation encoding="application/x-tex">P(\text{zero-gradient group}) = p^G + (1-p)^G</annotation></semantics></math></span><p>Plugging in numbers to see the trend (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span> is the per-sample accuracy; the table gives the probability that the group has zero gradient):</p>
<table>
  <thead>
      <tr>
          <th><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span></th>
          <th><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">G=8</annotation></semantics></math></span></th>
          <th><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span></th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>0.5</td>
          <td>0.8%</td>
          <td>≈0</td>
      </tr>
      <tr>
          <td>0.8</td>
          <td>16.8%</td>
          <td>≈0</td>
      </tr>
      <tr>
          <td>0.9</td>
          <td>43.0%</td>
          <td>0.1%</td>
      </tr>
      <tr>
          <td>0.95</td>
          <td>66.3%</td>
          <td>3.8%</td>
      </tr>
      <tr>
          <td>0.99</td>
          <td>92.3%</td>
          <td>52.6%</td>
      </tr>
  </tbody>
</table>
<p>How to read it: for medium-difficulty questions (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span> in the 0.3–0.7 range), even a small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is basically safe; what really consumes <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is <strong>questions at the difficulty boundary</strong> — at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>=</mo><mn>0.95</mn></mrow><annotation encoding="application/x-tex">p=0.95</annotation></semantics></math></span>, with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>8</mn></mrow><annotation encoding="application/x-tex">G=8</annotation></semantics></math></span> two-thirds of the groups are all-correct and gradient-free, and only <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span> pushes that down to 4%. And the training trajectory pushes questions precisely toward the boundaries: for questions the model has learned, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>→</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">p \to 1</annotation></semantics></math></span>; for questions it can&rsquo;t learn, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>→</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">p \to 0</annotation></semantics></math></span>; and the later in training, the fewer questions remain in the &ldquo;gradient-bearing range.&rdquo; This is one source of the &ldquo;training efficiency and stability&rdquo; problems that the DAPO paper describes in its motivation.</p>
<p>Faced with the same table, there are two remedies. One is to increase <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> and brute-force <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>p</mi><mi>G</mi></msup></mrow><annotation encoding="application/x-tex">p^G</annotation></semantics></math></span> down — with costs growing multiplicatively. The other is DAPO&rsquo;s <strong>Dynamic Sampling</strong>: instead of increasing <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>, simply rule that zero-gradient groups are not accepted. DAPO&rsquo;s objective includes a hard constraint (the s.t. condition of Equation 8 in the paper):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mn>0</mn><mo>&lt;</mo><mrow><mo fence="true">∣</mo><mo stretchy="false">{</mo><msub><mi>o</mi><mi>i</mi></msub><mo>∣</mo><mtext mathvariant="monospace">is_equivalent</mtext><mo stretchy="false">(</mo><mi>a</mi><mo separator="true">,</mo><msub><mi>o</mi><mi>i</mi></msub><mo stretchy="false">)</mo><mo stretchy="false">}</mo><mo fence="true">∣</mo></mrow><mo>&lt;</mo><mi>G</mi></mrow><annotation encoding="application/x-tex">0 &lt; \left|\{o_i \mid \texttt{is\_equivalent}(a, o_i)\}\right| &lt; G</annotation></semantics></math></span><p>That is, the number of correct responses in a group must lie strictly between 0 and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> — all-correct and all-wrong groups are filtered out and replaced through oversampling, guaranteeing that every prompt in the batch has an effective gradient. For the same effective-group coverage, the price of filtering is additive oversampling rather than a multiplicative increase in <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>.</p>
<p>But there is another layer of coupling between filtering and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> that is easy to miss: <strong>the filtering decision is itself based on an estimate from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> samples</strong>. At <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>4</mn></mrow><annotation encoding="application/x-tex">G=4</annotation></semantics></math></span>, &ldquo;all 4 wrong&rdquo; does not mean the model can&rsquo;t solve the question — a question with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>=</mo><mn>0.3</mn></mrow><annotation encoding="application/x-tex">p=0.3</annotation></semantics></math></span> has a 24% chance of getting all 4 wrong and being mistakenly thrown on the scrap heap. Pairing a small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> with dynamic sampling will misjudge a whole set of medium-difficulty questions that &ldquo;actually carry gradient value&rdquo; as dead groups; the misjudgment rate falls exponentially with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><msup><mo stretchy="false">)</mo><mi>G</mi></msup></mrow><annotation encoding="application/x-tex"> (1-p)^G </annotation></semantics></math></span>, the same table). So dynamic sampling does not eliminate <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>&rsquo;s coverage problem; it merely converts it from &ldquo;zero gradient&rdquo; into &ldquo;mistaken discards&rdquo; (this point is an inference; the DAPO paper does not discuss the misjudgment rate).</p>
<h2 id="role-three-the-quality-of-the-std-normalization-denominator">Role Three: The Quality of the std-Normalization Denominator</h2>
<p>GRPO&rsquo;s advantage is also divided by the group std: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>=</mo><mo stretchy="false">(</mo><msub><mi>r</mi><mi>i</mi></msub><mo>−</mo><mtext>mean</mtext><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mtext>std</mtext></mrow><annotation encoding="application/x-tex">\hat{A}_i = (r_i - \text{mean})/\text{std}</annotation></semantics></math></span>. This std is likewise an estimate from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> samples and is very noisy when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is small; for a uniform group the std is exactly 0, so implementations either add an eps or skip the group (setting a divide-by-zero trap).</p>
<p>Dr.GRPO&rsquo;s critique of the std (see the <a href="https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/">dedicated post</a>
) — that when the std of all-correct or all-wrong groups approaches zero, the advantage is inflated to unreasonable magnitudes — is at its worst with small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>: when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is small, the std is more likely to land on a small value, and extreme advantages show up frequently. Extreme advantages in turn push up how far the ratio moves, indirectly raising the clip fraction (this chain is an inference: abnormally large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span> → <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_\theta</annotation></semantics></math></span> moves more within a single step → <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">w_t</annotation></semantics></math></span> hits the clipping bound sooner). In other words, part of the claim that &ldquo;large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> stabilizes training&rdquo; is really not about the baseline but about the std — two ledgers that the literature often lumps together.</p>
<p>One more theoretical result worth knowing (arXiv:2502.18548, &ldquo;What is the Alignment Objective of GRPO?&rdquo;): the authors give a framework characterizing GRPO&rsquo;s stationary policies and point out that GRPO&rsquo;s way of aggregating preferences differs fundamentally from the logarithmic pooling of standard RLHF — <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> affects not only the variance of the estimate but also the fixed point of the normalized objective itself. This is analytical work, and the conditions under which its conclusions hold (stationarity, infinitely many steps) are some distance from engineering practice, but it is a reminder of one thing: understanding <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> purely as a &ldquo;sampling budget&rdquo; is incomplete.</p>
<h2 id="counterpoint-are-2-responses-enough">Counterpoint: Are 2 Responses Enough?</h2>
<p>The argument of 2510.00977 is worth restating on its own terms: GRPO&rsquo;s effectiveness comes from an <strong>implicit contrastive objective</strong> — comparing just two responses already yields a learning signal, and the group mean is merely a control variate for variance reduction that does not need to be a precise statistical estimate, so 2-GRPO with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">G=2</annotation></semantics></math></span> also works and, structurally, even ends up connecting to DPO. Their theoretical analysis and experiments (at small to medium scale) both support this conclusion.</p>
<p>Does this contradict DeepSeekMath&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span>? Not necessarily. Of the things a large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> buys, baseline precision is only one, and possibly the least important. The others are coverage of boundary-difficulty questions (the second role), the reliability of the filtering decision (the companion to dynamic sampling), and exploration coverage — a group needs at least one correct response for the model to have a &ldquo;good sample&rdquo; to learn from; for a hard question with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi><mo>=</mo><mn>0.1</mn></mrow><annotation encoding="application/x-tex">p=0.1</annotation></semantics></math></span>, there is an 81% chance at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>2</mn></mrow><annotation encoding="application/x-tex">G=2</annotation></semantics></math></span> that the group contains no positive sample, versus only 0.1% at <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span>. These gains depend strongly on model scale and on the task difficulty distribution: for a small model grinding through GSM8K-level problems, the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span> distribution is concentrated in the middle range, so a small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> costs little; for a large model attacking AIME-level problems, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span> is pinned around 0.1, every positive sample is precious, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is the exploration budget itself. Whether the 2-GRPO conclusion extends to large models in long-CoT settings is something nobody can answer yet (the paper&rsquo;s experiments are limited in scale; this is a boundary, not a criticism).</p>
<h2 id="the-cost-ledger-and-my-take">The Cost Ledger, and My Take</h2>
<p>First, the accounting: rollouts per training step = number of prompts × <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>. DeepSeekMath&rsquo;s configuration (batch 1024, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span>) gets through only 16 prompts per step — prompt diversity and response diversity are traded against each other on the same budget sheet. One more configuration detail deserves attention: DeepSeekMath performs only <strong>one</strong> policy update per exploration stage (strictly on-policy, §4.2), so the ratio is always 1 and clipping never triggers, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>&rsquo;s entire effect shows up in the baseline and in coverage; whereas with DAPO/GSPO-style multi-mini-batch updates, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> also indirectly affects the clip fraction through std noise. The same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> does not have the same marginal return in the two training regimes.</p>
<p>My take: <strong>there is no universally optimal <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>, because it spends money on three budget sheets at once</strong> — baseline precision (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>σ</mi><mi>r</mi></msub><mi mathvariant="normal">/</mi><msqrt><mi>G</mi></msqrt></mrow><annotation encoding="application/x-tex">\sigma_r/\sqrt{G}</annotation></semantics></math></span>, with the fastest-diminishing returns), effective-group coverage (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>p</mi><mi>G</mi></msup></mrow><annotation encoding="application/x-tex">p^G</annotation></semantics></math></span>, with the most durable returns at boundary difficulty), and the exploration budget (the probability that a group contains a positive sample, almost linear on hard questions). The practical default (8–16) is the balance point under &ldquo;a medium-difficulty distribution + dynamic sampling as a safety net&rdquo;; 64 was the naive choice of the DeepSeekMath era, before dynamic sampling and before any clip modifications; 2 is the theoretical limit from the &ldquo;contrastive objective&rdquo; perspective, which in engineering terms nobody has yet dared to use at production scale. Before picking <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>, first ask yourself which of the three sheets is the bottleneck: if the questions are too hard (the exploration budget is tight), increase <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> or add dynamic sampling; if the questions are too varied (prompt diversity is tight), give the budget to the number of prompts; if training is unstable, check the std and the clip fraction first, and don&rsquo;t rush to increase <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>.</p>
<p>The first four posts in this series happen to form a single thread: Clip-Higher fixes the token-level clipping bounds, GSPO changes the unit of the ratio, TIS plugs the numerical gap between training and inference, and this post goes back to the source of the signal — how to allocate the sampling budget. Every &ldquo;self-evident&rdquo; default configuration in GRPO, once taken apart, turns out to have a ledger of its own.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>There is no public systematic ablation of the optimal coupling between <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span>, the learning rate, and the clip <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span>; whether small <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> + a large learning rate can match the signal-to-noise ratio of a large <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi></mrow><annotation encoding="application/x-tex">G</annotation></semantics></math></span> is backed only by scattered folk wisdom;</li>
<li>No paper has quantified the actual impact on data efficiency of dynamic sampling&rsquo;s &ldquo;mistaken discard&rdquo; rate (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>1</mn><mo>−</mo><mi>p</mi><msup><mo stretchy="false">)</mo><mi>G</mi></msup></mrow><annotation encoding="application/x-tex">(1-p)^G</annotation></semantics></math></span>) on medium-difficulty questions;</li>
<li>Whether 2-GRPO holds for large models, long CoT, and boundary-difficulty tasks is one of the most worthwhile public replications to attempt right now.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>DeepSeekMath / original GRPO paper: <a href="https://arxiv.org/abs/2402.03300">arXiv:2402.03300</a>
; <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>G</mi><mo>=</mo><mn>64</mn></mrow><annotation encoding="application/x-tex">G=64</annotation></semantics></math></span>, batch 1024, and the single-step on-policy update are in §4.2</li>
<li>DAPO paper: <a href="https://arxiv.org/abs/2503.14476">arXiv:2503.14476</a>
; the Dynamic Sampling constraint is in Equation 8, the ±1 reward in Equation 7</li>
<li>It Takes Two: Your GRPO Is Secretly DPO: <a href="https://arxiv.org/abs/2510.00977">arXiv:2510.00977</a>
</li>
<li>What is the Alignment Objective of GRPO?: <a href="https://arxiv.org/abs/2502.18548">arXiv:2502.18548</a>
</li>
<li>RLHF Book, Chapter 6 (the baseline unbiasedness condition, the RLOO leave-one-out formula): <a href="https://github.com/natolambert/rlhf-book/blob/main/book/chapters/06-policy-gradients.md"><code>book/chapters/06-policy-gradients.md</code></a>
; teaching-code default <code>num_rollouts: 8</code>: <a href="https://github.com/natolambert/rlhf-book/tree/main/code/policy_gradients/configs"><code>code/policy_gradients/configs/</code></a>
</li>
<li>The zero-gradient-group probability table, the mistaken discards, and the clip-fraction chain are this post&rsquo;s own calculations/inferences, each flagged where it appears in the text</li>
<li>Previous post: <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">The Ratio&rsquo;s Wrong Unit: GSPO Goes Sequence-Level</a>
</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>One Model, Two Probabilities: Cost of Mismatch</title>
      <link>https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/</link>
      <pubDate>Thu, 27 Aug 2026 13:00:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/</guid>
      <description>Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.</description>
      <content:encoded><![CDATA[<p>Rollouts run on an inference engine and training runs on a training engine, and the same model&rsquo;s numbers don&rsquo;t match across the two: the distribution you actually sample from deviates from the policy you assume, and nominally on-policy training quietly becomes off-policy. This post breaks down where the discrepancy comes from, its two consequences (biased gradients and a deployment gap), the lineage of algorithmic patches, and Sea AI Lab&rsquo;s conclusion tracing the root cause to BF16 precision.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/tis-train-inference-mismatch/">interactive report</a>
 (in Chinese), with a drill-down diagram of the two-engine mechanism, a comparison of the patch lineage, and a bit-field breakdown of the precision formats.</p>
</blockquote>
<h2 id="the-same-hard-to-explain-collapse">The Same Hard-to-Explain Collapse</h2>
<p>In the second half of 2025, several teams independently reported the same phenomenon: early in GRPO training everything looks normal and reward climbs steadily; then, at some point, it abruptly turns downward until training collapses. There is no gradient spike before the collapse, nothing is wrong with the hyperparameters, and if you rerun with a different batch of training queries, the collapse still comes, just at a different time.</p>
<p>The August 2025 technical report by Defog (Feng Yao, ByteDance Seed), “Your Efficient RL Framework Secretly Brings You Off-Policy RL Training”, offered an explanation that surprised quite a few people: <strong>you think you are doing on-policy training, but the distribution your rollout data actually comes from has already drifted away from the policy you are optimizing</strong>. The problem is not in the algorithm but in the numerics—the probabilities computed by the engine that generates the data don&rsquo;t match those computed by the engine that trains on it.</p>
<h2 id="why-one-model-computes-two-probabilities">Why One Model Computes Two Probabilities</h2>
<p>Modern LLM RL frameworks are split into two halves. The rollout side uses an inference engine (vLLM, SGLang), heavily optimized at the kernel level for high throughput; the training side uses a training engine (Megatron, PyTorch FSDP, DeepSpeed), which computes each token&rsquo;s log probability and the gradients. Both sides load the same weights, so in theory they should be the same policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span>.</p>
<p>But because their engineering implementations differ, the engines don&rsquo;t compute exactly the same results. Sea AI Lab&rsquo;s FP16 paper (arXiv:2510.26788) summarizes where the discrepancies come from:</p>
<ul>
<li><strong>Different computation patterns</strong>. Inference generates autoregressively, token by token; training runs a parallel forward pass over the whole sequence under teacher forcing. For the same token string, the different computation processes use different floating-point reduction orders, and errors accumulate differently.</li>
<li><strong>Different kernel implementations</strong>. The inference engine&rsquo;s RMSNorm, RoPE, and attention kernels are specially tuned for latency and throughput, and differ numerically, by tiny amounts, from the training-side implementations.</li>
<li><strong>Different parallel sharding</strong>. Training and inference use different tensor-parallel and data-parallel sharding, and a different order of cross-GPU reductions changes the floating-point result.</li>
<li><strong>MoE top-k gating is sensitive to small changes</strong>. Expert selection is sensitive to tiny fluctuations in the logits, so if the same weights are run once through each of the two engines, the selected experts can differ. The 10% expert drift that the <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">GSPO post</a>
 mentions occurs along the temporal dimension of “before vs. after an update”; here we mean the spatial dimension of “same weights, two engines”. Gating sensitivity introduces a discrepancy along each of the two dimensions separately.</li>
<li><strong>The precision format&rsquo;s mantissa is too short</strong>. The default BF16 has only 7 mantissa bits, so none of the tiny discrepancies cited above has enough numerical precision headroom, and they end up producing noticeable deviations. This point is the crux of the whole post; more on it later.</li>
</ul>
<p>So the inference engine actually corresponds to <strong>a numerically different policy</strong> from the training engine&rsquo;s, denoted <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span>. Data is sampled from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span>, but the gradient is computed under the assumption of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span>. The boundary between on-policy and off-policy ultimately comes down to the floating-point behavior of kernels.</p>
<h2 id="two-consequences-biased-gradients-discounted-deployment">Two Consequences: Biased Gradients, Discounted Deployment</h2>
<p>Consequence one: <strong>biased gradients</strong>. The theoretical policy gradient requires sampling from the policy being optimized, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi mathvariant="script">J</mi><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi mathvariant="double-struck">E</mi><mrow><mi>y</mi><mo>∼</mo><mi>π</mi><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><mi>θ</mi><mo stretchy="false">)</mo></mrow></msub><mrow><mo fence="true">[</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><mi>π</mi><mo stretchy="false">(</mo><mi>y</mi><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>⋅</mo><mi>R</mi><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mi>y</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\nabla_\theta\mathcal{J}(x,\theta)=\mathbb{E}_{y\sim\pi(\cdot|x,\theta)}\left[\nabla_\theta\log\pi(y|x,\theta)\cdot R(x,y)\right]</annotation></semantics></math></span><p><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi><mo stretchy="false">(</mo><mi>y</mi><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><mi>θ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi(y|x,\theta)</annotation></semantics></math></span> is the likelihood of the response in the training forward pass, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mi>y</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">R(x,y)</annotation></semantics></math></span> is the reward, and the expectation has to be taken under the distribution of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span>. But the actual samples come from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><msub><mi mathvariant="script">J</mi><mrow><mi>b</mi><mi>i</mi><mi>a</mi><mi>s</mi><mi>e</mi><mi>d</mi></mrow></msub><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi mathvariant="double-struck">E</mi><mrow><mi>y</mi><mo>∼</mo><mi>μ</mi><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><mi>θ</mi><mo stretchy="false">)</mo></mrow></msub><mrow><mo fence="true">[</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><mi>π</mi><mo stretchy="false">(</mo><mi>y</mi><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>⋅</mo><mi>R</mi><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mi>y</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow><mo mathvariant="normal">≠</mo><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi mathvariant="script">J</mi><mo stretchy="false">(</mo><mi>x</mi><mo separator="true">,</mo><mi>θ</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\nabla_\theta\mathcal{J}_{biased}(x,\theta)=\mathbb{E}_{y\sim\mu(\cdot|x,\theta)}\left[\nabla_\theta\log\pi(y|x,\theta)\cdot R(x,y)\right]\neq \nabla_\theta\mathcal{J}(x,\theta)</annotation></semantics></math></span><p>The distribution underlying the expectation has suddenly changed, and the equality no longer holds. The bias is usually small, but RL is an iterative process that keeps pushing the distribution in one direction, so small differences between <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span> get amplified step by step over training. The paper also observes a phenomenon that can serve as an early-warning signal: in every run that eventually collapsed, the gap between <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span> kept widening before the collapse; in extreme cases, under the same weights, the same token had a probability approaching 1 in one engine and approaching 0 in the other.</p>
<p>Consequence two: <strong>the deployment gap</strong>. Even if the gradient bias in training is fully corrected, the optimization target is still the training engine&rsquo;s probability distribution, while deployment and evaluation run on the inference engine:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>arg</mi><mo>⁡</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>θ</mi></munder><msub><mi mathvariant="double-struck">E</mi><mrow><mi>y</mi><mo>∼</mo><mi>μ</mi></mrow></msub><mi>R</mi><mo mathvariant="normal">≠</mo><mi>arg</mi><mo>⁡</mo><munder><mrow><mi>max</mi><mo>⁡</mo></mrow><mi>θ</mi></munder><msub><mi mathvariant="double-struck">E</mi><mrow><mi>y</mi><mo>∼</mo><mi>π</mi></mrow></msub><mi>R</mi></mrow><annotation encoding="application/x-tex">\arg\max_\theta \mathbb{E}_{y\sim\mu}R \neq \arg\max_\theta \mathbb{E}_{y\sim\pi}R</annotation></semantics></math></span><p>Parameters that are optimal under <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span> are not necessarily optimal under <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span>. This gap is visible in the paper&rsquo;s experiments: the most stable algorithmic patch under BF16 (sequence-level masked correction) reaches a peak training accuracy of 95% and an AIME 2024 score of 34%; under FP16, the numbers are 99% and 39%. On this measure, all the algorithmic patches make zero difference—they only address the bias in the training process and cannot address this deployment gap.</p>
<h2 id="algorithmic-patches-and-their-limits">Algorithmic Patches and Their Limits</h2>
<p>At its core, the problem is “sampling from <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span> but taking the expectation under <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span>”, and the textbook fix is an importance-sampling correction: multiply each sample by the train-inference probability ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi><mi mathvariant="normal">/</mi><mi>μ</mi></mrow><annotation encoding="application/x-tex">\pi/\mu</annotation></semantics></math></span>. The improvements in the second half of 2025 came mainly in two steps:</p>
<p><strong>Patch one: token-level truncated correction (Defog)</strong>. For each token, compute the ratio of the training probability to the inference probability, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ρ</mi><mi>t</mi></msub><mo>=</mo><mi>π</mi><mo stretchy="false">(</mo><msub><mi>y</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mo>&lt;</mo><mi>t</mi></mrow></msub><mo separator="true">,</mo><msup><mi>θ</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mi>μ</mi><mo stretchy="false">(</mo><msub><mi>y</mi><mi>t</mi></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mo>&lt;</mo><mi>t</mi></mrow></msub><mo separator="true">,</mo><msup><mi>θ</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\rho_t = \pi(y_t|x,y_{&lt;t},\theta&#x27;)/\mu(y_t|x,y_{&lt;t},\theta&#x27;)</annotation></semantics></math></span>, truncate it at an upper bound <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi></mrow><annotation encoding="application/x-tex">C</annotation></semantics></math></span> (3 in the experiments), and multiply it into each term of the GRPO gradient. Note that the denominator <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span> is the inference engine&rsquo;s probability, so the training engine has to run one more forward pass over the same sample to obtain the numerator, adding about 25% training overhead (estimated assuming the backward pass costs roughly 2× the forward pass). Result: the collapse is delayed, but training still collapses after peaking at 82% (VeRL) / 88% (Oat).</p>
<p><strong>Patch two: sequence-level masked correction</strong> (“When Speed Kills Stability”, Liu et al.). The authors point out that the token-level correction yields a biased gradient, and change the correction to compute a single ratio for the entire sequence, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ρ</mi><mo>=</mo><mi>π</mi><mo stretchy="false">(</mo><mi>y</mi><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msup><mi>θ</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mi>μ</mi><mo stretchy="false">(</mo><mi>y</mi><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msup><mi>θ</mi><mo mathvariant="normal" lspace="0em" rspace="0em">′</mo></msup><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\rho = \pi(y|x,\theta&#x27;)/\mu(y|x,\theta&#x27;)</annotation></semantics></math></span>, masking out the gradient of the whole sequence whenever it exceeds <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>C</mi></mrow><annotation encoding="application/x-tex">C</annotation></semantics></math></span>. The idea is in the same lineage as GSPO (although GSPO&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> serves only as the quantity being optimized, whereas <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ρ</mi></mrow><annotation encoding="application/x-tex">\rho</annotation></semantics></math></span> here is a correction factor). It is the only algorithmic patch that does not collapse under BF16; the price is high variance in the sequence-level ratio and slow convergence, and the deployment gap remains exactly as it was—the peak is only 95%/34%.</p>
<p>What the two patches have in common: both accept the fact that <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi><mo mathvariant="normal">≠</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">\mu \neq \pi</annotation></semantics></math></span> and compensate sample by sample, at the cost of extra compute and added variance, and neither can solve the deployment gap.</p>
<h2 id="the-root-cause-bf16s-7-bit-mantissa">The Root Cause: BF16&rsquo;s 7-Bit Mantissa</h2>
<p>Rather than pushing further down the patching path, the Sea AI Lab paper went straight to examining the numerical format.</p>
<p>BF16 and FP16 both use 16 bits but allocate them differently. BF16 uses 8 exponent bits + 7 mantissa bits, giving it a dynamic range on par with FP32: pretraining doesn&rsquo;t have to worry about overflow and doesn&rsquo;t need loss scaling, which is why it became the default. FP16 uses 5 exponent bits + 10 mantissa bits: its dynamic range is much smaller (it relies on loss scaling to prevent gradient underflow), but adjacent representable numbers are 8× as dense as in BF16 (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>10</mn></msup></mrow><annotation encoding="application/x-tex">2^{10}</annotation></semantics></math></span> vs. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mn>2</mn><mn>7</mn></msup></mrow><annotation encoding="application/x-tex">2^7</annotation></semantics></math></span>), so its precision is higher.</p>
<p>The paper&rsquo;s assessment: the requirements change in the RL fine-tuning stage. The dynamic range of weights and activations has already been settled during pretraining, so BF16&rsquo;s FP32-level range no longer serves any practical purpose here, while every kernel discrepancy between training and inference needs FP16&rsquo;s 3 extra mantissa bits to provide enough numerical precision headroom. Once precision is unified to FP16, the two engines&rsquo; outputs agree numerically on the vast majority of tokens, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span> coincide again. Offline analysis shows that under BF16 the sequence-level train-inference mismatch grows exponentially with generation length, while FP16 reduces the mismatch substantially, to about 1/24 of BF16&rsquo;s. In the precision ablations, the “BF16 training + FP32 inference” combination is also fully stable, but inference is 3× slower; using FP16 throughout is the best combination of stability and efficiency.</p>
<p>The most telling result is the next set of experiments: <strong>with FP16, the performance differences between algorithms almost disappear</strong>. GRPO, token-level TIS, GSPO, and sequence-level MIS, which differ markedly under BF16, end up close to one another under FP16, and even the plainest importance-sampling policy gradient, without any correction, achieves the best performance. The reason is not hard to see: once the gap between <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>π</mi></mrow><annotation encoding="application/x-tex">\pi</annotation></semantics></math></span> is close to zero, training returns to an approximately on-policy regime, and every algorithm meant to “correct the mismatch” loses the thing it was correcting. (One more detail: on VeRL, GSPO&rsquo;s gradients turn into NaN after 1,200 steps, which suggests that algorithm-level improvements may be of no use while the numerical problem remains unsolved.) The engineering routes cited here bear this out too: the SGLang team hand-aligned the kernel implementations of the two compute paths (the November 2025 Unified FP8 work), and Miles/slime aligned training and inference in one go at the system level. Both are effective, but they require deep kernel expertise and a heavy engineering investment, and they are hard to port. The numerical format is the cheapest of all the solutions: it only takes changing a few lines of configuration, while keeping the existing loss scaling mechanism.</p>
<h2 id="reading-it-alongside-the-previous-two-posts">Reading It Alongside the Previous Two Posts</h2>
<p>The three posts in this series happen to approach the same problem at four levels:</p>
<ul>
<li><strong>DAPO Clip-Higher</strong>: assumes training and inference are consistent, and adjusts the clip bounds inside the algorithm to fix entropy collapse;</li>
<li><strong>GSPO</strong>: moves the optimization objective to sequence-level units, is naturally robust to the discrepancy in <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span>, and also tolerates MoE routing drift;</li>
<li><strong>TIS algorithmic patches</strong> (the first half of this post): accept the fact that <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi><mo mathvariant="normal">≠</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">\mu \neq \pi</annotation></semantics></math></span> and apply importance-sampling compensation sample by sample;</li>
<li><strong>FP16 and kernel alignment</strong> (the second half of this post): make <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>μ</mi><mo>≈</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">\mu \approx \pi</annotation></semantics></math></span>, eliminating the mismatch directly.</li>
</ul>
<p>Seen against this arc, GSPO §5.4&rsquo;s observation that “sequence-level likelihoods are highly tolerant of precision differences between training and inference” is no longer just a side benefit—it is a direct algorithm-level response to the train-inference mismatch (TIS) problem.</p>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>in engineering terms, on-policy is a numerical problem</strong>. The algorithm&rsquo;s definition says “sample from the current policy”; in engineering practice you can only sample from “a numerical approximation of the current policy under some kernel implementation”. Once the two differ by a certain magnitude, the premises of the theoretical analysis no longer hold, and existing alerting mechanisms cannot raise an alarm about it: the training curve looks normal, the loss looks normal, and only the logprob gap between the training side and the inference side is silently diverging. The practical advice is straightforward: make the train-inference logprob gap (or KL) a routine monitoring metric. Its cost is close to zero, and the early-warning value the paper demonstrates is empirically measured.</p>
<p>Second, <strong>BF16&rsquo;s default status deserves to be questioned, but the FP16 conclusion should not be over-generalized</strong>. The evidence covers 1.5B to 30B MoE, two frameworks, LoRA, and two model families, which is solid; the paper itself also states that it does not claim universality: using FP16 on very large models requires managing overflow, and if the industry moves toward FP8 (with only 3–4 mantissa bits), the train-inference mismatch problem will reappear. The current FP8 RL route (LMSYS, NVIDIA) unifies training and inference on FP8 rather than going back to FP16. The precision problem will either be taken seriously again or resurface in another form.</p>
<p>Third, <strong>the deployment gap is the most direct and least studied problem behind all this</strong>. Although the gradient bias can be corrected, and the train-inference mismatch can be eliminated by aligning both the kernels and the precision format on the two sides, the problem that “the trained parameters are not optimal for the deployment engine” has not disappeared; it has only shrunk. When evaluation switches to the inference engine, which policy does the reported score actually belong to? With mixed multi-engine deployment, how should the gap be computed? So far I have not seen a systematic study.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>Whether the small train-inference discrepancies that remain under FP16 will accumulate back into a problem over ultra-long training runs of thousands of steps;</li>
<li>The final performance of the unified-FP8 route versus the FP16 route under the same compute budget: there is currently no public head-to-head comparison;</li>
<li>The deployment gap will still appear, only smaller: when evaluating with an engine different from the training environment, which policy should the score be attributed to?</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Sea AI Lab / NUS: <a href="https://arxiv.org/abs/2510.26788">Defeating the Training-Inference Mismatch via FP16</a>
 (the source of this post&rsquo;s experiments, formulas, and data)</li>
<li>Defog (Feng Yao et al.): “Your Efficient RL Framework Secretly Brings You Off-Policy RL Training”, 2025-08, published on Notion; the original report could not be accessed directly due to technical limitations, so its token-level correction formula is paraphrased from §2.1 of the FP16 paper</li>
<li>Liu et al.: “When Speed Kills Stability: Demystifying RL Collapse from the Inference-Training Mismatch”, 2025, published on Notion; sequence-level masked correction, paraphrased in the same way</li>
<li>Engineering routes: <a href="https://www.lmsys.org/blog/2025-11-25-fp8-rl/">Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL</a>
 (LMSYS); Miles/slime: All-In-One Solution to Training-Inference Mismatch</li>
<li>Textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
, Appendix B.1: two engineering treatments of the old logprobs produced by the rollout engine (use the results returned by the inference side directly, or recompute them on the training side and check their accuracy)</li>
<li>Previous posts: <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
 and <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">The Ratio&rsquo;s Wrong Unit: GSPO Goes Sequence-Level</a>
</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>The Ratio&#39;s Wrong Unit: GSPO Goes Sequence-Level</title>
      <link>https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/</link>
      <pubDate>Wed, 26 Aug 2026 17:40:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/</guid>
      <description>Qwen says GRPO&amp;rsquo;s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.</description>
      <content:encoded><![CDATA[<p>GRPO carries over PPO&rsquo;s token-level importance ratio, which the Qwen team deems a fundamental misuse of importance sampling that leads to irreversible collapse in long-response MoE training. GSPO lifts the ratio, clipping, and optimization all to the sequence level: it clips two orders of magnitude more tokens, yet trains more efficiently. This post unpacks that chain of argument, along with the premises and costs the paper leaves unstated.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/gspo-sequence-level-importance-sampling/">interactive report</a>
 (in Chinese), with drill-down formula breakdowns, a clipping-band comparison, and a dilution-effect calculator.</p>
</blockquote>
<h2 id="the-problem-a-patch-exposed">The Problem a Patch Exposed</h2>
<p>Before mid-2025, training MoE models with GRPO required a dedicated patch.</p>
<p>In the GSPO paper (arXiv:2507.18071), the Qwen team describes the approach they had been using: <strong>Routing Replay</strong>. During sampling, the expert routes activated by the old policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}}</annotation></semantics></math></span> are cached; when computing the importance ratio, these routes are &ldquo;replayed&rdquo; on the new policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_\theta</annotation></semantics></math></span>, so that the ratio&rsquo;s numerator and denominator run through the same activated network. The patch was needed because of one number they observed: in the 48-layer Qwen3-30B-A3B, after each gradient update, roughly 10% of the experts activated for the same rollout sample under the new policy differ from those under the old policy. Once the routing changes, the per-token probabilities change with it, the token-level importance ratios fluctuate violently, and GRPO simply fails to converge.</p>
<p>The patch works, but its costs are real: extra GPU memory and communication overhead, and forcing the new policy onto the old routes amounts to capping the model&rsquo;s effective capacity. What is more troubling is something else — even with the patch, GRPO on long-response tasks with large models still suffers <strong>irreversible model collapse</strong>. The paper&rsquo;s wording is strong: once collapse occurs, rolling back to a checkpoint, carefully tuning the clipping range, extending the generation length, switching RL queries — none of it can bring the model back.</p>
<p>The token-level importance ratio has been standard since the PPO era, in use for eight years. Why has it suddenly turned pathological in RL for LLMs? GSPO&rsquo;s answer: it did not suddenly fall ill; it was used in the wrong place from the start — this is a misuse of importance sampling, and in the era of small models, short responses, and near-on-policy training, the error was simply masked.</p>
<h2 id="why-importance-sampling-fails">Why Importance Sampling &ldquo;Fails&rdquo;</h2>
<p>Let me first write down importance sampling (IS) itself. We want to use samples from a behavior distribution <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mrow><mi>b</mi><mi>e</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\pi_{beh}</annotation></semantics></math></span> to estimate the expectation, under a target distribution <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mrow><mi>t</mi><mi>a</mi><mi>r</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\pi_{tar}</annotation></semantics></math></span>, of a function <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="double-struck">E</mi><mrow><mi>z</mi><mo>∼</mo><msub><mi>π</mi><mrow><mi>t</mi><mi>a</mi><mi>r</mi></mrow></msub></mrow></msub><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><msub><mi mathvariant="double-struck">E</mi><mrow><mi>z</mi><mo>∼</mo><msub><mi>π</mi><mrow><mi>b</mi><mi>e</mi><mi>h</mi></mrow></msub></mrow></msub><mrow><mo fence="true">[</mo><mfrac><mrow><msub><mi>π</mi><mrow><mi>t</mi><mi>a</mi><mi>r</mi></mrow></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><mrow><msub><mi>π</mi><mrow><mi>b</mi><mi>e</mi><mi>h</mi></mrow></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow></mfrac><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\mathbb{E}_{z\sim\pi_{tar}}[f(z)] = \mathbb{E}_{z\sim\pi_{beh}}\left[\frac{\pi_{tar}(z)}{\pi_{beh}(z)} f(z)\right]</annotation></semantics></math></span><p>This identity does not hold by virtue of the ratio itself; it relies on <strong>averaging over many samples</strong>: only when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>≫</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">N \gg 1</annotation></semantics></math></span> do the fluctuations of the random ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>w</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi>π</mi><mrow><mi>t</mi><mi>a</mi><mi>r</mi></mrow></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><msub><mi>π</mi><mrow><mi>b</mi><mi>e</mi><mi>h</mi></mrow></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">w(z)=\pi_{tar}(z)/\pi_{beh}(z)</annotation></semantics></math></span> average out, so that the estimate converges to the true value. Multiplying a single sample by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>w</mi></mrow><annotation encoding="application/x-tex">w</annotation></semantics></math></span> provides no correction whatsoever; it merely multiplies <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(z)</annotation></semantics></math></span> by a high-variance random number.</p>
<p>Now look at what GRPO does. It computes a ratio for every token:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">w_{i,t}(\theta)=\frac{\pi_\theta(y_{i,t}|x,y_{i,&lt;t})}{\pi_{\theta_{old}}(y_{i,t}|x,y_{i,&lt;t})}</annotation></semantics></math></span><p>Here <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">y_{i,t}</annotation></semantics></math></span> is the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span>-th response&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span>-th token, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">y_{i,&lt;t}</annotation></semantics></math></span> is the context preceding it. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}}(\cdot|x,y_{i,&lt;t})</annotation></semantics></math></span> is the next-token distribution at step <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span>, and <strong>only one sample</strong> is drawn from it: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">y_{i,t}</annotation></semantics></math></span>. In other words, every ratio uses a single sample to &ldquo;correct&rdquo; a single-step distribution — by the criterion above, this does not constitute an IS correction at all; it merely injects a per-token random weight into the gradient. The longer the sequence, the more random weights accumulate; the larger and sparser the model (MoE), the higher the variance of each individual weight. Clipping then zeroes out the gradients of tokens that fall outside the range, and the noise is amplified into &ldquo;randomly wiping out the learning signal of a subset of tokens&rdquo;. Chapter 6 of the RLHF Book gives two concrete descriptions of the failure as it plays out: a single token with a large ratio can dominate the entire update; and tokens within the same response are clipped independently of one another, shredding the learning signal.</p>
<p>The GSPO paper distills the crux into a single sentence: <strong>the unit of the optimization objective should match the unit of the reward</strong>. In RLVR, the reward is assigned to the entire sequence (1 point for a correct answer, 0 for a wrong one), whereas GRPO&rsquo;s correction and clipping both happen at the token level. With the units mismatched, the more &ldquo;conscientious&rdquo; the correction, the more precisely targeted the damage.</p>
<p>To be honest, I should add one point here: GRPO&rsquo;s formulation is not without theoretical grounding. Treating each token as one action step and decomposing the IS ratio of the whole trajectory into per-step ratios is the standard derivation path in RL. The problem lies at the implementation level, where the per-token ratios are used one by one as independent correction weights and clipped one by one — an approximation that stops being harmless under large vocabularies, long trajectories, and single samples. This is the GSPO paper&rsquo;s critical angle, not an axiom; readers are free to reach their own judgment.</p>
<h2 id="gspo-moving-ratio-clipping-and-optimization-to-the-sequence-level">GSPO: Moving Ratio, Clipping, and Optimization to the Sequence Level</h2>
<p>The GSPO objective:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="script">J</mi><mrow><mi>G</mi><mi>S</mi><mi>P</mi><mi>O</mi></mrow></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi mathvariant="double-struck">E</mi><mrow><mi>x</mi><mo>∼</mo><mi mathvariant="script">D</mi><mo separator="true">,</mo><mo stretchy="false">{</mo><msub><mi>y</mi><mi>i</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo>∼</mo><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub></mrow></msub><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mi>min</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo separator="true">,</mo><mtext> </mtext><mrow><mi mathvariant="normal">c</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">p</mi></mrow><mo stretchy="false">(</mo><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo separator="true">,</mo><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo fence="true">)</mo></mrow><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\mathcal{J}_{GSPO}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\{y_i\}_{i=1}^G\sim\pi_{\theta_{old}}}\left[\frac{1}{G}\sum_{i=1}^G \min\left(s_i(\theta)\hat{A}_i,\ \mathrm{clip}(s_i(\theta),1-\varepsilon,1+\varepsilon)\hat{A}_i\right)\right]</annotation></semantics></math></span><p>The difference from GRPO is visible at a glance: the per-token sum <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><msub><mo>∑</mo><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\frac{1}{|y_i|}\sum_t</annotation></semantics></math></span> outside the min-clip is gone, and each response as a whole occupies just one term in the objective. The advantage is still the group-normalized <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\hat{A}_i</annotation></semantics></math></span> (standardized with the mean and standard deviation of the rewards within the group), shared by the entire response.</p>
<p>The core is the definition of the sequence-level ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msup><mrow><mo fence="true">(</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo stretchy="false">)</mo></mrow><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo stretchy="false">)</mo></mrow></mfrac><mo fence="true">)</mo></mrow><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac></msup><mo>=</mo><mi>exp</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><mi>log</mi><mo>⁡</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow></mfrac><mo fence="true">)</mo></mrow></mrow><annotation encoding="application/x-tex">s_i(\theta)=\left(\frac{\pi_\theta(y_i|x)}{\pi_{\theta_{old}}(y_i|x)}\right)^{\frac{1}{|y_i|}}=\exp\left(\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\log\frac{\pi_\theta(y_{i,t}|x,y_{i,&lt;t})}{\pi_{\theta_{old}}(y_{i,t}|x,y_{i,&lt;t})}\right)</annotation></semantics></math></span><p>This is the geometric mean of the whole-sequence likelihood ratio, equivalent to taking the arithmetic mean of the per-token log-ratios and then exponentiating. The length normalization <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> cannot be dropped: without it, thousands of ratios get multiplied together, likelihood changes in just a few tokens can make the overall ratio swing violently, and responses of different lengths would need different clipping ranges — there would be no way to set <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> at all. After normalization, the numerical range of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> is decoupled from sequence length, and a single <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> covers everything. (The hands-on-modern-rl textbook omits this <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> from its formula when introducing GSPO in Section 16.4; go by Equation 7 in the paper — leave it out and the whole chain of argument breaks.)</p>
<p>The meaning of clipping changes accordingly. In GRPO, clipping zeroes out the gradient of an individual token — the signal is chopped up, but the sample remains; in GSPO, clipping removes an entire response from the gradient estimate — a sample is either used in full or not at all. This aligns, in granularity, with &ldquo;the reward is given to the whole sequence&rdquo;.</p>
<h2 id="the-gradient-view-unequal-vs-equal-weighting">The Gradient View: Unequal vs. Equal Weighting</h2>
<p>Expand the gradients of the two objectives (Equations 8–12 in the paper), and the fundamental difference comes down to one thing: what multiplies each token&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\nabla_\theta\log\pi_\theta</annotation></semantics></math></span>.</p>
<p>In GRPO&rsquo;s gradient, token <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span>&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∇</mi><mi>log</mi><mo>⁡</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">\nabla\log\pi</annotation></semantics></math></span> is multiplied by <strong>its own ratio</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">w_{i,t}</annotation></semantics></math></span>. After the min-clip, these per-token weights lie in <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">(0, 1+\varepsilon]</annotation></semantics></math></span> (when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A}_i&gt;0</annotation></semantics></math></span>) or <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">[</mo><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mo>+</mo><mi mathvariant="normal">∞</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">[1-\varepsilon, +\infty)</annotation></semantics></math></span> (when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>&lt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A}_i&lt;0</annotation></semantics></math></span>) — within the same response, the gradient weights that different tokens receive can differ by orders of magnitude, and as they accumulate over training steps, the consequences are hard to predict.</p>
<p>In GSPO&rsquo;s gradient, all tokens of the same response share <strong>the same scalar weight</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><msub><mi mathvariant="script">J</mi><mrow><mi>G</mi><mi>S</mi><mi>P</mi><mi>O</mi></mrow></msub><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>⋅</mo><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\nabla_\theta\mathcal{J}_{GSPO} = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^G s_i(\theta)\hat{A}_i \cdot \frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \nabla_\theta\log\pi_\theta(y_{i,t}|x,y_{i,&lt;t})\right]</annotation></semantics></math></span><p>The tokens&rsquo; contributions to the update differ only in the magnitude of their own <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∇</mi><mi>log</mi><mo>⁡</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">\nabla\log\pi</annotation></semantics></math></span>; there is no longer any per-token random weight modulating them. The paper describes this as eliminating the destabilizing factor in GRPO.</p>
<h2 id="evidence-three-experimental-facts">Evidence: Three Experimental Facts</h2>
<p>The paper&rsquo;s experimental setup is itself worth noting: a cold-start model (obtained by SFT from Qwen3-30B-A3B-Base), with each rollout batch split into 4 mini-batches for gradient updates — a standard off-policy setting, and precisely the setting in which, as the previous post on Clip-Higher put it, &ldquo;clipping actually starts to do its job&rdquo; (only then does the ratio deviate from 1). The GRPO baseline&rsquo;s clipping range of 0.2/0.27 was carefully tuned; GSPO uses 3e-4 / 4e-4.</p>
<p>Fact one: <strong>the three-orders-of-magnitude gap in ε is not a typo; it is an inevitable consequence of averaging</strong>. A single token-level ratio can easily shoot up to 1.2; the sequence-level ratio is the exp of the average of the per-token log-ratios, and averaging cancels out positive and negative fluctuations, so <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> naturally stays close to 1, fluctuating only slightly. This can be worked out in detail (what follows is my own estimate; the paper does not report these statistics): suppose that after one mini-batch update, each token&rsquo;s log-ratio drift has, on average, a standard deviation of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math></span> (say, 5%), the response length is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span>, and the drifts of different tokens are approximately independent; then the standard deviation of the averaged drift shrinks to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mi mathvariant="normal">/</mi><msqrt><mi>L</mi></msqrt></mrow><annotation encoding="application/x-tex">\sigma/\sqrt{L}</annotation></semantics></math></span> — for <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo>=</mo><mn>2000</mn></mrow><annotation encoding="application/x-tex">L=2000</annotation></semantics></math></span>, a reduction to about <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mn>45</mn></mrow><annotation encoding="application/x-tex">1/45</annotation></semantics></math></span>. Give such a nearly motionless ratio a clipping band of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>±</mo><mn>0.2</mn></mrow><annotation encoding="application/x-tex">\pm 0.2</annotation></semantics></math></span> and it will never touch the edges; clipping might as well not exist. So GSPO has to tighten the band to a few parts in ten thousand: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.2</mn><mi mathvariant="normal">/</mi><mn>3</mn><mo>×</mo><msup><mn>10</mn><mrow><mo>−</mo><mn>4</mn></mrow></msup><mo>≈</mo><mn>667</mn></mrow><annotation encoding="application/x-tex">0.2 / 3\times10^{-4} \approx 667</annotation></semantics></math></span> — that is where the three-orders-of-magnitude narrowing comes from. Put another way: a tolerance band of 3e-4 corresponds to allowing an average log drift of only about 0.03% per token. I should also be upfront about how sound this assumption is: in real training, per-token drifts are not independent (a single update raises or lowers the probabilities of similar contexts en masse), so the actual reduction factor falls short of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msqrt><mi>L</mi></msqrt></mrow><annotation encoding="application/x-tex">\sqrt{L}</annotation></semantics></math></span> — but this only makes the conclusion of Fact two below stronger.</p>
<p>Fact two: <strong>a hundred times more tokens get clipped, yet efficiency is higher</strong>. The band is a thousand times narrower, yet a hundred times more tokens are clipped — this sounds contradictory, but once you multiply two factors together, it isn&rsquo;t (the estimates below are, as above, my own). First factor: relative to their respective fluctuation scales, the band is actually narrower. The fluctuation shrinks by only about 45×, while the band shrinks by 667×; using each ratio&rsquo;s own fluctuation as the yardstick, GSPO&rsquo;s band is about 15× narrower than GRPO&rsquo;s (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msqrt><mi>L</mi></msqrt><mi mathvariant="normal">/</mi><mn>667</mn></mrow><annotation encoding="application/x-tex">\sqrt{L}/667</annotation></semantics></math></span>), so <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> goes out of bounds far more easily than <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">w_t</annotation></semantics></math></span> does. Plugging in concrete numbers (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mo>=</mo><mn>5</mn><mi mathvariant="normal">%</mi></mrow><annotation encoding="application/x-tex">\sigma=5\%</annotation></semantics></math></span>, approximately Gaussian drift): GRPO&rsquo;s 0.2 corresponds to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>4</mn><mi>σ</mi></mrow><annotation encoding="application/x-tex">4\sigma</annotation></semantics></math></span>, so the probability of a single token going out of bounds is a few in ten thousand; GSPO&rsquo;s 3e-4 corresponds to only about <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.27</mn><mi>σ</mi></mrow><annotation encoding="application/x-tex">0.27\sigma</annotation></semantics></math></span> (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.0003</mn><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mn>0.05</mn><mi mathvariant="normal">/</mi><mn>45</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">0.0003/(0.05/45)</annotation></semantics></math></span>), so each response has a better-than-even chance of being thrown out in its entirety. Second factor: going out of bounds is charged to the whole response. In GRPO, a token that goes out of bounds discards only itself; in GSPO, a single out-of-bounds event removes the entire span of several thousand tokens. Multiply the two factors, and the total number of clipped tokens comes to roughly a hundred times GRPO&rsquo;s — the gap in clipping fractions measured in Figure 2 of the paper is indeed about two orders of magnitude, so the magnitudes line up. The truly counterintuitive part is the second half of the claim: fewer tokens are used for training, yet training efficiency and benchmark performance are better (winning on all three curves: AIME'24 / LiveCodeBench / CodeForces). The paper&rsquo;s interpretation: GRPO&rsquo;s token-level gradient estimates are noisy to begin with, and the portion that happens to survive is no more useful than the portion that gets clipped; GSPO&rsquo;s sequence-level filtering provides a more reliable learning signal — fewer but clean beats more but dirty.</p>
<p>Fact three: <strong>MoE no longer needs Routing Replay</strong>. The sequence likelihood <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi_\theta(y_i|x)</annotation></semantics></math></span> is insensitive to routing jitter — when 10% of the experts change, individual token probabilities may shift noticeably, but the likelihood of the whole sequence (a reflection of the model&rsquo;s language-modeling capability) does not change drastically. GSPO cares only about the latter, so simply computing the ratio as usual is enough for it to converge; the patch&rsquo;s memory and communication overhead is saved entirely, and the model is no longer bound to the old routes.</p>
<p>There is also an infrastructure dividend on the side (§5.4 of the paper): in practice, the training engine (Megatron) and the inference engines (SGLang, vLLM) differ in numerical precision, so GRPO has to recompute the old policy&rsquo;s likelihoods with the training engine before it can safely use them; sequence-level likelihoods are far more tolerant of precision discrepancies, so GSPO can optimize directly with the likelihoods returned by the inference engine and skip the recomputation. For partial rollouts, multi-turn RL, and architectures that decouple training from inference, this is a tangible throughput dividend.</p>
<h2 id="appendix-how-the-3e-44e-4-clipping-range-was-set">Appendix: How the 3e-4/4e-4 Clipping Range Was Set</h2>
<p>At the level of the paper: it isn&rsquo;t stated. The GSPO paper gives only the result (left bound 3e-4, right bound 4e-4), with no selection procedure and no ablation on the clipping range. Why this value rather than 1e-4 or 1e-3 is impossible for outsiders to know; a reasonable guess is that the authors found it by trial and error on their own setup (cold-start Qwen3-30B-A3B, each batch split into 4 mini-batches), but that is only a guess.</p>
<p>At the level of principle (what follows is my inference, not the paper&rsquo;s content): the clipping range&rsquo;s job is to let normal fluctuations through and stop abnormal ones, so it should be set at a small multiple of the typical fluctuation magnitude of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>ε</mi><mo>≈</mo><mi>k</mi><mo>⋅</mo><mfrac><mi>σ</mi><msqrt><mi>L</mi></msqrt></mfrac></mrow><annotation encoding="application/x-tex">\varepsilon \approx k \cdot \frac{\sigma}{\sqrt{L}}</annotation></semantics></math></span><p>Here <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math></span> is the magnitude of the per-token log-ratio drift after one update, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span> is the response length, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span> is the multiple of the fluctuation you want to let through. Plugging in the illustrative numbers from earlier (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mo>=</mo><mn>5</mn><mi mathvariant="normal">%</mi></mrow><annotation encoding="application/x-tex">\sigma=5\%</annotation></semantics></math></span>, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo>=</mo><mn>2000</mn></mrow><annotation encoding="application/x-tex">L=2000</annotation></semantics></math></span>) gives <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mi mathvariant="normal">/</mi><msqrt><mi>L</mi></msqrt><mo>≈</mo><mn>0.0011</mn></mrow><annotation encoding="application/x-tex">\sigma/\sqrt{L} \approx 0.0011</annotation></semantics></math></span>, so the paper&rsquo;s 3e-4 corresponds to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>≈</mo><mn>0.27</mn></mrow><annotation encoding="application/x-tex">k \approx 0.27</annotation></semantics></math></span> — letting through only about a quarter of the typical fluctuation, a very strict filter, which squares with the measured &ldquo;tens of percent of tokens clipped&rdquo;.</p>
<p>This framework also shows that 3e-4 cannot be copied as-is; it is coupled to three things: the degree of off-policyness (the staler the data, the larger <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math></span>, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> must be loosened accordingly, or else every sample gets clipped away); response length (the longer <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span>, the smaller the fluctuation after averaging, so the same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> becomes stricter — suitable values for 32k and 2k differ by about 4×); and the desired clipping strength (there is no public data on the optimal clipping fraction, so a reproduction has to run its own sweep). Change the configuration, and this number has to be set again.</p>
<h2 id="my-three-reservations">My Three Reservations</h2>
<p>First, <strong>the &ldquo;single-sample&rdquo; critique applies equally to GSPO itself</strong>. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> is also a ratio computed from a single sample <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">y_i</annotation></semantics></math></span>, likewise without any <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>≫</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">N \gg 1</annotation></semantics></math></span> averaging. So GSPO&rsquo;s improvement should not be understood as &ldquo;restoring the theoretical correctness of importance sampling&rdquo; — it does no such thing; the paper&rsquo;s IS argument is intuitive, not formal. What it really does is bring three things together: unit alignment (what gets clipped is an entire &ldquo;bad sample&rdquo; rather than random tokens), variance normalization (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y|</annotation></semantics></math></span> squeezes the numerical range onto a common scale), and equal-weight gradients (eliminating the accumulation of unequal per-token weights). All three are engineering arguments — solid, but not theorems.</p>
<p>Second, <strong>the geometric mean is a diluter, and that cuts both ways</strong>. In a 10,000-token CoT, if the probability of one critical token drifts by 50%, its contribution to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> is only <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mrow><mi>log</mi><mo>⁡</mo><mn>1.5</mn><mi mathvariant="normal">/</mi><mn>10000</mn></mrow></msup><mo>≈</mo><mn>1.00004</mn></mrow><annotation encoding="application/x-tex">e^{\log 1.5 / 10000} \approx 1.00004</annotation></semantics></math></span> — nearly invisible. Against precision noise between the training and inference engines, dilution is a feature (this is exactly where the infrastructure dividend comes from); for detecting &ldquo;a critical token altered by a policy update&rdquo;, dilution is a loophole. GSPO works properly precisely because it is paired with an extremely narrow clipping band on the order of three parts in ten thousand — making trust-region judgments based on average drift rather than single-point drift. Whether this combination goes blind to certain failure modes on ultra-long CoTs (for example, the policy deviating sharply on only a short stretch of reasoning, which gets let through after averaging) is something the paper does not answer.</p>
<p>Third, <strong>sequence-level clipping gives up on making use of &ldquo;partially correct&rdquo; samples</strong>. If a response gets 90% of its reasoning right but miscalculates in the final step, GSPO either uses it whole or throws it out whole. The paper&rsquo;s way out is the GSPO-token variant: using stop-gradient, it combines the sequence-level weight <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> (taking only its value, without backpropagating gradients through it) with the per-token term <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub><mi mathvariant="normal">/</mi><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\pi_\theta / \mathrm{sg}[\pi_\theta]</annotation></semantics></math></span>; the resulting objective, clipping condition, and theoretical gradient are fully equivalent to GSPO&rsquo;s, but advantages can be customized per token — an opening left for multi-turn RL and fine-grained rewards. At the other end of this line is VAPO: bring the value model back outright to do token-level credit assignment. Sequence-level vs. token-level is not a dispute over right and wrong, but two industrial answers to the old question of &ldquo;credit-assignment granularity&rdquo;.</p>
<h2 id="how-it-relates-to-clip-higher-and-a-detail-no-one-has-pointed-out">How It Relates to Clip-Higher, and a Detail No One Has Pointed Out</h2>
<p>The previous post covered DAPO&rsquo;s Clip-Higher: the pathology is the same — a structural flaw of token-level clipping under off-policy training. DAPO&rsquo;s remedy is to adjust the bounds (raising the upper bound from 0.2 to 0.28 to cut exploratory tokens some slack); GSPO&rsquo;s remedy is to change the unit (the whole response shares a single ratio). The two are not mutually exclusive — Qwen&rsquo;s practice is precisely GSPO stacked with a Dr.GRPO-style bias correction that removes the normalization.</p>
<p>There is one more detail worth pointing out: the GSPO paper&rsquo;s clipping range of 3e-4 / 4e-4 is itself <strong>asymmetric</strong> — the right bound is wider than the left, leaving 33% more room in the reinforcing direction (the ratio breaking upward when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A}&gt;0</annotation></semantics></math></span>). Isn&rsquo;t that just a sequence-level version of Clip-Higher? The paper does not discuss where this asymmetry comes from, but its direction is consistent with DAPO&rsquo;s logic: let good samples go a little further. Two teams independently prescribed remedies pointing in the same direction for the same pathology — that says more than any single paper&rsquo;s ablation could.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>The method for choosing <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> has itself not been disclosed: 3e-4/4e-4 comes with results but no ablation, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi><mo>≈</mo><mi>k</mi><mo>⋅</mo><mi>σ</mi><mi mathvariant="normal">/</mi><msqrt><mi>L</mi></msqrt></mrow><annotation encoding="application/x-tex">\varepsilon \approx k \cdot \sigma/\sqrt{L}</annotation></semantics></math></span> is only this post&rsquo;s inferred framework; the optimal range of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span>, and the effect of combining it with clip-higher-style asymmetric bounds, all call for public reproduction;</li>
<li>Whether the dilution effect of the geometric mean systematically lets &ldquo;abrupt local shifts&rdquo; slip through on ultra-long CoTs (beyond 32k tokens) needs to be verified with purpose-built experiments;</li>
<li>The actual effectiveness of GSPO-token paired with turn-level rewards in multi-turn agentic RL is, for now, backed only by Qwen&rsquo;s internal data.</li>
</ul>
<hr>
<blockquote>
<p><strong>Update log (2026-08-29)</strong>: Expanded based on feedback from three rounds of reader Q&amp;A — the &ldquo;Evidence&rdquo; section now includes the full derivation of the clipping-band arithmetic (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mi mathvariant="normal">/</mi><msqrt><mi>L</mi></msqrt></mrow><annotation encoding="application/x-tex">\sigma/\sqrt{L}</annotation></semantics></math></span> variance reduction, the 667× narrowing, the band being ~15× narrower in relative terms, and the out-of-bounds probability calculation); a new section, &ldquo;Appendix: How the 3e-4/4e-4 Clipping Range Was Set&rdquo;, was added (the paper does not disclose its selection method; an inferred framework and three couplings); the first open question was rewritten accordingly. All derivations are labeled estimates, not content from the paper.</p>
</blockquote>
<h2 id="references">References</h2>
<ul>
<li>GSPO paper: <a href="https://arxiv.org/abs/2507.18071">arXiv:2507.18071v2</a>
; the equation numbers and experimental data in this post all come from its v2</li>
<li>RLHF Book, Chapter 6 (importance sampling background; GSPO/CISPO subsections): <a href="https://rlhfbook.com/">rlhfbook.com</a>
; <a href="https://github.com/natolambert/rlhf-book/blob/main/book/chapters/06-policy-gradients.md"><code>book/chapters/06-policy-gradients.md</code></a>
</li>
<li>hands-on-modern-rl, Section 16.4 (comparison of the GRPO family of improvements; note that its GSPO formula omits the length normalization): <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter18_grpo/grpo-family.md"><code>docs/chapter18_grpo/grpo-family.md</code></a>
</li>
<li>Previous post: <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>Entropy Collapse: What DAPO&#39;s Clip-Higher Fixes</title>
      <link>https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/</link>
      <pubDate>Wed, 26 Aug 2026 14:30:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/</guid>
      <description>Naive GRPO collapses entropy fast. DAPO&amp;rsquo;s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.</description>
      <content:encoded><![CDATA[<p>When naive GRPO is used to reproduce R1, entropy collapses rapidly and samples within a group converge; DAPO&rsquo;s Clip-Higher merely raises the upper clipping bound from 0.2 to 0.28. Behind that one number lies a structural asymmetry of PPO-Clip on the probability axis. This post unpacks the failure chain, the mechanism, the evidence, and the limits.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/dapo-clip-higher-entropy-collapse/">interactive report</a>
 (in Chinese), with drill-down breakdowns of the formulas and charts of the ablation data.</p>
</blockquote>
<h2 id="a-scene-from-a-training-run">A Scene from a Training Run</h2>
<p>In early 2025, many teams were doing the same thing: reproducing DeepSeek-R1&rsquo;s training with GRPO. The DAPO paper (arXiv:2503.14476) reports the authors&rsquo; own starting point—running naive GRPO on the Qwen2.5-32B base model scored only 30 on AIME 2024, while DeepSeek&rsquo;s R1-Zero-Qwen-32B scored 47.</p>
<p>The gap did not open up gradually; it was a very specific way of dying: during training, the actor model&rsquo;s entropy drops rapidly, the responses sampled as a group for the same problem grow more and more alike, and in the end they are nearly identical. The paper calls this <strong>entropy collapse</strong>. Sampling loses its diversity, rewards within the group converge, GRPO&rsquo;s advantages all approach zero, and the gradient signal extinguishes itself. Training is still running, but it can no longer learn anything.</p>
<p>The first remedy DAPO prescribes is called <strong>Clip-Higher</strong>: raise PPO-Clip&rsquo;s upper clipping bound ε from the default 0.2 to 0.28 and leave the lower bound at 0.2. The change is a single number. Why can this one number stop the collapse? The answer lies in a structural property of PPO-Clip that is rarely spelled out.</p>
<h2 id="first-pinning-down-entropy">First, Pinning Down &ldquo;Entropy&rdquo;</h2>
<p>This whole post revolves around a single word—&ldquo;entropy&rdquo;—so let me pin down its definition first. The entropy here is exactly the entropy of information theory, and the object it applies to is <strong>the probability distribution over the entire vocabulary that the model outputs each time it generates a token</strong>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>H</mi><mo>=</mo><mo>−</mo><munder><mo>∑</mo><mrow><mi>x</mi><mo>∈</mo><mtext>vocab</mtext></mrow></munder><mi>p</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mi>log</mi><mo>⁡</mo><mi>p</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">H = -\sum_{x \in \text{vocab}} p(x) \log p(x)</annotation></semantics></math></span><p>The flatter the distribution (many tokens with similar probabilities), the higher the entropy, and the more the model &ldquo;can&rsquo;t make up its mind&rdquo;; the sharper the distribution (one token takes 0.99 and the rest split 0.01), the lower the entropy, and the more &ldquo;certain&rdquo; the model is. A numerical example: suppose the vocabulary has only 3 tokens; the distribution [0.9, 0.05, 0.05] has an entropy of about 0.39 nat, while the uniform distribution [1/3, 1/3, 1/3] has about 1.10 nat. Sample 10 times from the former and you get the same token 9 times; samples from the latter spread out evenly—<strong>entropy is the mathematical measure of &ldquo;sampling diversity&rdquo;</strong>. The &ldquo;actor model entropy&rdquo; that the DAPO paper monitors is the average of these per-step distribution entropies over rollout generation.</p>
<p>Why it is the lifeline of RL training: all of GRPO&rsquo;s training signal comes from the within-group reward variance, and that variance comes from sampling diversity. With high entropy, 16 responses sampled for the same problem all differ, some right and some wrong, and the advantages are nonzero; when entropy collapses, the 16 converge, the rewards are all identical, and the advantages are all zero.</p>
<p>Two caveats, so as not to over-read this: first, <strong>higher entropy is not always better</strong>—excessively high entropy corresponds to runaway modes such as gibberish and repeated words; the paper only says entropy should be kept within an &ldquo;appropriate range&rdquo;, without saying how to determine that range (one of the open questions at the end of this post). Second, <strong>don&rsquo;t conflate entropy with sampling temperature</strong>: entropy is the intrinsic shape of the distribution, while temperature is how you draw from it; it takes parameters like temperature 1.0 at rollout time to cash entropy out as diversity. Clip-Higher affects the former—it keeps training itself from squeezing the distribution ever sharper.</p>
<h2 id="first-working-out-the-ppo-clip-math">First, Working Out the PPO-Clip Math</h2>
<p>The PPO/GRPO objective, viewed token by token, is:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mi>min</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><msub><mi>r</mi><mi>t</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mtext> </mtext><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub><mo separator="true">,</mo><mtext> clip</mtext><mo stretchy="false">(</mo><msub><mi>r</mi><mi>t</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo separator="true">,</mo><mtext> </mtext><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mtext> </mtext><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">)</mo><mtext> </mtext><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub><mo fence="true">)</mo></mrow><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">J(\theta) = \mathbb{E}\left[\min\left(r_t(\theta)\,\hat{A}_t,\ \text{clip}(r_t(\theta),\ 1-\varepsilon,\ 1+\varepsilon)\,\hat{A}_t\right)\right]</annotation></semantics></math></span><p>The symbols, one by one:</p>
<ul>
<li><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>r</mi><mi>t</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>o</mi><mi>t</mi></msub><mo>∣</mo><mi>q</mi><mo separator="true">,</mo><msub><mi>o</mi><mrow><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><msub><mi>o</mi><mi>t</mi></msub><mo>∣</mo><mi>q</mi><mo separator="true">,</mo><msub><mi>o</mi><mrow><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">r_t(\theta) = \pi_\theta(o_t \mid q, o_{&lt;t}) / \pi_{\theta_{old}}(o_t \mid q, o_{&lt;t})</annotation></semantics></math></span>, the <strong>importance sampling ratio</strong>: the current policy&rsquo;s probability for the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span>-th token divided by the old policy&rsquo;s probability at sampling time. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>=</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">r=1</annotation></semantics></math></span> means the policy has not changed.</li>
<li><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\hat{A}_t</annotation></semantics></math></span>, the <strong>advantage</strong>: how much better this token is than average. In GRPO it is computed by within-group normalization: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo>=</mo><mo stretchy="false">(</mo><msub><mi>R</mi><mi>i</mi></msub><mo>−</mo><mtext>mean</mtext><mo stretchy="false">(</mo><mo stretchy="false">{</mo><mi>R</mi><mo stretchy="false">}</mo><mo stretchy="false">)</mo><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><mtext>std</mtext><mo stretchy="false">(</mo><mo stretchy="false">{</mo><mi>R</mi><mo stretchy="false">}</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\hat{A}_{i,t} = (R_i - \text{mean}(\{R\})) / \text{std}(\{R\})</annotation></semantics></math></span>.</li>
<li><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span>, the <strong>clipping range</strong>: how far the ratio is allowed to deviate from 1; the default in the vast majority of implementations is 0.2—a default that comes from the Atari and robotic-control experiments in the original 2017 PPO paper.</li>
</ul>
<p>On where the advantage comes from, PPO and GRPO take two different paths; don&rsquo;t mix them up when reading the literature:</p>
<ul>
<li><strong>PPO</strong>: relies on a learned value function (the critic) and estimates the advantage token by token with GAE: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>t</mi></msub><mo>=</mo><msub><mo>∑</mo><mi>l</mi></msub><mo stretchy="false">(</mo><mi>γ</mi><mi>λ</mi><msup><mo stretchy="false">)</mo><mi>l</mi></msup><msub><mi>δ</mi><mrow><mi>t</mi><mo>+</mo><mi>l</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\hat{A}_t = \sum_{l}(\gamma\lambda)^l \delta_{t+l}</annotation></semantics></math></span>, where <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>δ</mi><mi>t</mi></msub><mo>=</mo><msub><mi>r</mi><mi>t</mi></msub><mo>+</mo><mi>γ</mi><mi>V</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mrow><mi>t</mi><mo>+</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo><mo>−</mo><mi>V</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mi>t</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)</annotation></semantics></math></span>. If the critic is accurate, the advantage has low variance; if the critic is off, the whole training run goes off with it.</li>
<li><strong>GRPO</strong>: no critic; it samples G responses for the same problem and normalizes with the within-group rewards. In essence it uses the group mean as the baseline—this is REINFORCE&rsquo;s variance-reduction trick, except that the baseline doesn&rsquo;t have to be learned; it is computed directly from the samples in the same group.</li>
</ul>
<p>The GRPO route has three direct consequences, all of which come up later. First, <strong>identical rewards across a group yield all zeros</strong>: when all G responses are right or all are wrong, the mean equals each response&rsquo;s own reward, every advantage is 0, and the problem contributes no gradient at all. Second, <strong>dividing by std carries an implicit weighting</strong>: groups with a small std (say, only 1 of 16 correct) get their weight amplified by the normalization, and the difficulty bias that Dr.GRPO criticizes comes from this term. Third, <strong>the advantage is computed per whole response and shared by every token in it</strong>: in a wrong response, the first 80% of the reasoning steps may well be correct, yet they receive the same negative advantage as the final step where the calculation went wrong—GRPO by itself has no token-level credit assignment.</p>
<p>How the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>min</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\min</annotation></semantics></math></span> behaves depends on the sign of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span>—this is the key to understanding everything:</p>
<ul>
<li><strong><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &gt; 0</annotation></semantics></math></span> (good token, to be reinforced)</strong>: when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>&gt;</mo><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">r &gt; 1+\varepsilon</annotation></semantics></math></span>, the clipped branch is selected, the objective becomes a constant, and <strong>the gradient is zero</strong>. In other words, each update step can raise a token&rsquo;s probability to at most <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">1+\varepsilon</annotation></semantics></math></span> times its value at sampling time; beyond that, no gradient is given.</li>
<li><strong><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&lt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &lt; 0</annotation></semantics></math></span> (bad token, to be suppressed)</strong>: symmetrically, each step can push the probability down at most to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>−</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">1-\varepsilon</annotation></semantics></math></span> times its value at sampling time.</li>
</ul>
<p>There is an easily misread point here that is worth unpacking: <strong>for an advantage of either sign, the clip is one-sided</strong>. Take the case <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &gt; 0</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>&lt;</mo><mn>1</mn><mo>−</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">r &lt; 1-\varepsilon</annotation></semantics></math></span>: the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>min</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\min</annotation></semantics></math></span> still selects <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>⋅</mo><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">r \cdot \hat{A}</annotation></semantics></math></span> (because it is the smaller one). Although the objective&rsquo;s value is small, <strong>the gradient is alive</strong>—<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>d</mi><mi>J</mi><mi mathvariant="normal">/</mi><mi>d</mi><mi>π</mi><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">dJ/d\pi &gt; 0</annotation></semantics></math></span>, and the optimizer still pulls this good token&rsquo;s probability back up with the full slope <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span>. A small value does not mean suppression: <strong>the value is the outcome; the gradient is the action</strong>. What the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>min</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\min</annotation></semantics></math></span> does is <strong>withdraw an incentive</strong>, never <strong>apply a reverse incentive</strong>. So the complete structure is: for good tokens, only the incentive to rise is capped (the plateau on the right); for bad tokens, only the incentive to suppress is capped (the plateau on the left); correction in the opposite direction always goes through with the full gradient.</p>
<p>The core mechanism chart in the interactive report is drawn on exactly this logic. How to read it: the horizontal axis is the ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span>, the vertical axis is this token&rsquo;s contribution to the objective, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">J(r)</annotation></semantics></math></span>, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover></mrow><annotation encoding="application/x-tex">\hat{A}</annotation></semantics></math></span> is normalized to 1. For <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>&lt;</mo><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">r &lt; 1+\varepsilon</annotation></semantics></math></span>, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>J</mi><mo stretchy="false">(</mo><mi>r</mi><mo stretchy="false">)</mo><mo>=</mo><mi>r</mi></mrow><annotation encoding="application/x-tex">J(r)=r</annotation></semantics></math></span> is a rising straight line (the gradient is alive); once <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mo>&gt;</mo><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">r &gt; 1+\varepsilon</annotation></semantics></math></span>, it becomes a horizontal plateau (the gradient stops there).</p>
<p>Chapter 6 of the RLHF Book also makes a point of noting that the clip is not always at work. Only when the ratio actually goes out of bounds and the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>min</mi><mo>⁡</mo></mrow><annotation encoding="application/x-tex">\min</annotation></semantics></math></span> happens to select the clipped branch is the gradient cut off. The clip fraction is the metric that monitors exactly this. Keep this in mind; it will come in handy later.</p>
<h2 id="symmetric-clipping-asymmetric-effects">Symmetric Clipping, Asymmetric Effects</h2>
<p>Now for some arithmetic. Let <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi><mo>=</mo><mn>0.2</mn></mrow><annotation encoding="application/x-tex">\varepsilon = 0.2</annotation></semantics></math></span> and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &gt; 0</annotation></semantics></math></span>, and consider two tokens:</p>
<ul>
<li>A token with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo>=</mo><mn>0.9</mn></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}} = 0.9</annotation></semantics></math></span>: the upper bound on its probability is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.9</mn><mo>×</mo><mn>1.2</mn><mo>=</mo><mn>1.08</mn></mrow><annotation encoding="application/x-tex">0.9 \times 1.2 = 1.08</annotation></semantics></math></span>. A probability cannot exceed 1, so <strong>it is effectively unconstrained</strong>—in one round it can be pushed from 0.9 to above 0.99.</li>
<li>A token with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo>=</mo><mn>0.01</mn></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}} = 0.01</annotation></semantics></math></span>: its upper bound is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.01</mn><mo>×</mo><mn>1.2</mn><mo>=</mo><mn>0.012</mn></mrow><annotation encoding="application/x-tex">0.01 \times 1.2 = 0.012</annotation></semantics></math></span>. <strong>An absolute increment of 0.002</strong>—it barely moves.</li>
</ul>
<p>First, let me pin down &ldquo;one round&rdquo; precisely: it means one &ldquo;sample → update&rdquo; cycle. After sampling, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}}</annotation></semantics></math></span> is frozen as a snapshot, and then multiple gradient updates are run on this batch of data (16 for DAPO, paper §4.1); during this time <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_\theta</annotation></semantics></math></span> gradually drifts away from the snapshot and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span> starts drifting from 1. Once <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span> crosses <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mo>+</mo><mi>ε</mi></mrow><annotation encoding="application/x-tex">1+\varepsilon</annotation></semantics></math></span>, that token&rsquo;s gradient is cut off, and the remaining updates in this round no longer push it. So the precise meaning of &ldquo;an upper bound of 0.012&rdquo; is <strong>the point where the incentive vanishes</strong>, not a hard clamp—with a large step, the probability may overshoot it in a single step; it&rsquo;s just that once past it, the incentive drops to zero: nothing pushes it further, and nothing pulls it back.</p>
<p>And the budget <strong>refreshes every round</strong>: the next round resamples, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}}</annotation></semantics></math></span> is updated, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi></mrow><annotation encoding="application/x-tex">r</annotation></semantics></math></span> resets to 1, and the token gets another <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>×</mo><mn>1.2</mn></mrow><annotation encoding="application/x-tex">\times 1.2</annotation></semantics></math></span> budget. So a low-probability token grows along a multiplicative path: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.01</mn><mo>→</mo><mn>0.012</mn><mo>→</mo><mn>0.0144</mn><mo>→</mo><mo>⋯</mo></mrow><annotation encoding="application/x-tex">0.01 \to 0.012 \to 0.0144 \to \cdots</annotation></semantics></math></span>—exponential growth in ratio terms, yet in absolute terms it climbs very slowly through the first stretch; a high-probability token hits 1.0 and saturates within a round or two. The suppression direction is symmetric but harsher: with <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&lt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A} &lt; 0</annotation></semantics></math></span>, each round can cut the probability down to as little as <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>×</mo><mn>0.8</mn></mrow><annotation encoding="application/x-tex">\times 0.8</annotation></semantics></math></span>, leaving 0.33 after five rounds, so low-probability tokens are quickly beaten down to near 0—<strong>multiplicative decay on the way down is much faster than multiplicative growth on the way up</strong>.</p>
<p>This is exactly where the problem lies: <strong>clipping is measured on the &ldquo;ratio&rdquo; scale, but learning takes effect on the &ldquo;probability&rdquo; axis.</strong> With the same 20% tolerance on the ratio, a high-probability token gets close to 0.2 of room in absolute probability, while a low-probability token gets 0.002. The ranges of movement of the two kinds of token differ by nearly a hundredfold.</p>
<p>And &ldquo;exploration&rdquo; in RL is carried precisely by low-probability tokens—tokens the current policy is not inclined to pick but that may lead to new solutions. DAPO&rsquo;s measurements bear this out: the tokens actually clipped by the upper bound during training have an average probability below 0.2 (Figure 3a of the paper). In other words, <strong>the tokens the clipping mechanism keeps pinning down are precisely the exploration tokens</strong>; exploitation tokens (tokens the model is already very sure about) are almost never constrained.</p>
<p>Long-CoT settings amplify this bias: a reasoning chain runs to thousands of tokens, an exploration token at a critical turning point can move only 0.2% per round, and before it has a chance to grow, the rest of the distribution has already sharpened further.</p>
<h2 id="the-closed-loop-of-collapse">The Closed Loop of Collapse</h2>
<p>Put this micro-level bias into GRPO&rsquo;s training dynamics and you get a complete failure chain:</p>
<ol>
<li>Exploration tokens&rsquo; probabilities can&rsquo;t grow, high-probability tokens keep climbing, the distribution sharpens, and <strong>entropy drops</strong>;</li>
<li>A sharper distribution means the responses sampled as a group (say, 16) for the same problem become <strong>more and more alike</strong>;</li>
<li>GRPO&rsquo;s advantage is computed by within-group normalization; as within-group rewards converge, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mtext>std</mtext><mo>→</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\text{std} \to 0</annotation></semantics></math></span> and all advantages approach zero;</li>
<li><strong>The gradient signal vanishes</strong>; the model loses its direction of improvement and can only keep reinforcing patterns it is already sure of—back to step 1.</li>
</ol>
<p>In the RLVR setting this loop is especially lethal, because its entire training signal is the within-group reward variance. The reward is a rule-based 0/1 (+1 for a correct answer, -1 for a wrong one), with no reward model to provide a continuous signal. Once diversity dries up, the signal source is cut off. Entropy collapse is not &ldquo;a monitoring metric that looks bad&rdquo;; it is training burning its own fuel.</p>
<p>This also explains the logic behind the ordering of DAPO&rsquo;s four techniques: Clip-Higher prevents the collapse, while Dynamic Sampling (filtering out prompts whose samples are all correct or all wrong) stanches the bleeding once the collapse has already produced zero-gradient groups. One treats the root cause, the other the symptoms; the paper puts Clip-Higher in the first section.</p>
<h2 id="clip-higher-raising-only-the-upper-bound-is-deliberate">Clip-Higher: Raising Only the Upper Bound Is Deliberate</h2>
<p>What Clip-Higher does is split the symmetric clip into two hyperparameters:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mtext>clip</mtext><mrow><mo fence="true">(</mo><msub><mi>r</mi><mi>t</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo separator="true">,</mo><mtext> </mtext><mn>1</mn><mo>−</mo><msub><mi>ε</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub><mo separator="true">,</mo><mtext> </mtext><mn>1</mn><mo>+</mo><msub><mi>ε</mi><mrow><mi>h</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub><mo fence="true">)</mo></mrow><mo separator="true">,</mo><mspace width="1em"/><msub><mi>ε</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub><mo>=</mo><mn>0.2</mn><mo separator="true">,</mo><mtext> </mtext><msub><mi>ε</mi><mrow><mi>h</mi><mi>i</mi><mi>g</mi><mi>h</mi></mrow></msub><mo>=</mo><mn>0.28</mn></mrow><annotation encoding="application/x-tex">\text{clip}\left(r_t(\theta),\ 1-\varepsilon_{low},\ 1+\varepsilon_{high}\right),\quad \varepsilon_{low}=0.2,\ \varepsilon_{high}=0.28</annotation></semantics></math></span><p>The upper bound goes from 1.2 to 1.28, so a low-probability exploration token&rsquo;s room to rise in a single round goes from +20% to +28%; compounded by the multiplicative accumulation of the budget across rounds, the effect is considerable.</p>
<p>The question worth asking is: why not touch the lower bound? The paper is blunt about it—<strong>if <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>ε</mi><mrow><mi>l</mi><mi>o</mi><mi>w</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\varepsilon_{low}</annotation></semantics></math></span> were loosened (say, allowing probabilities to be pushed down to 0.6×), the gradient from negative advantages would, by the same ratio logic, push low-probability tokens straight down to near 0</strong>; those tokens would effectively be deleted from the sampling space, and the collapse would only come faster. We did the multiplicative-decay math earlier: the downward direction is already faster than the upward one, so clipping in the suppression direction should, if anything, be tightened, not loosened.</p>
<p>So Clip-Higher is not some vague move like &ldquo;opening the door to exploration a little wider&rdquo;; it is a precise piece of asymmetric surgery: <strong>loosen the restraints on the way up, stay on guard on the way down</strong>.</p>
<p>In terms of results, Figure 2 of the paper shows that after Clip-Higher is applied, the entropy curve clearly recovers and AIME accuracy rises in step. The ablation chain (Qwen2.5-32B, AIME 2024 avg@32):</p>
<table>
  <thead>
      <tr>
          <th>Configuration</th>
          <th>AIME score</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Naive GRPO</td>
          <td>30</td>
      </tr>
      <tr>
          <td>+ Overlong Filtering</td>
          <td>36</td>
      </tr>
      <tr>
          <td>+ Clip-Higher</td>
          <td>38</td>
      </tr>
      <tr>
          <td>+ Soft Overlong Punishment</td>
          <td>41</td>
      </tr>
      <tr>
          <td>+ Token-level Loss</td>
          <td>42</td>
      </tr>
      <tr>
          <td>+ Dynamic Sampling (full DAPO)</td>
          <td>50</td>
      </tr>
  </tbody>
</table>
<p>To be honest: Clip-Higher on its own contributes only +2 points, not the largest gain in the chain (Overlong Filtering gives +6, and adding Dynamic Sampling at the end gives +8). But its role isn&rsquo;t to add points—what it fixes is the stall point of &ldquo;whether training can keep scaling&rdquo;. Once entropy collapses, the three techniques that follow have nothing to work with.</p>
<h2 id="two-premises-rarely-spelled-out">Two Premises Rarely Spelled Out</h2>
<p><strong>Premise one: the clip activates only when training is off-policy.</strong> Chapter 6 of the RLHF Book points out that in RL practice for language models, many algorithms take only one gradient step per batch—in that case the ratio is identically 1, the clip never triggers, and what actually acts as the regularizer is the KL penalty. DAPO, by contrast, takes 16 gradient steps per rollout (mini-batch 512, paper §4.1): the policy is updated multiple times within a batch, and only then does the ratio deviate from 1 and the clip actually start working. <strong>So the value of the Clip-Higher knob is coupled to how off-policy the training is</strong>: if your training is strictly single-step on-policy, this knob simply does not exist. Conversely, asynchronous RL (where rollout and training overlap and samples are inherently off-policy) is the soil in which it really comes into play.</p>
<p><strong>Premise two: DAPO also drops the KL term.</strong> First, let me be clear about the KL penalty itself. The KL divergence measures how far the current policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_\theta</annotation></semantics></math></span> is from the reference policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mrow><mi>r</mi><mi>e</mi><mi>f</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\pi_{ref}</annotation></semantics></math></span> (usually the frozen SFT/base model):</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>D</mi><mrow><mi>K</mi><mi>L</mi></mrow></msub><mo stretchy="false">(</mo><msub><mi>π</mi><mi>θ</mi></msub><mi mathvariant="normal">∥</mi><msub><mi>π</mi><mrow><mi>r</mi><mi>e</mi><mi>f</mi></mrow></msub><mo stretchy="false">)</mo><mo>=</mo><msub><mi mathvariant="double-struck">E</mi><mrow><mi>x</mi><mo>∼</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow></msub><mrow><mo fence="true">[</mo><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo>−</mo><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mrow><mi>r</mi><mi>e</mi><mi>f</mi></mrow></msub><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">D_{KL}(\pi_\theta \| \pi_{ref}) = \mathbb{E}_{x \sim \pi_\theta}\left[\log \pi_\theta(x) - \log \pi_{ref}(x)\right]</annotation></semantics></math></span><p>Classic RLHF puts it into the objective as a penalty term (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>−</mo><mi>β</mi><msub><mi>D</mi><mrow><mi>K</mi><mi>L</mi></mrow></msub></mrow><annotation encoding="application/x-tex">-\beta D_{KL}</annotation></semantics></math></span>), for two reasons: first, <strong>to prevent reward hacking</strong>—the reward model is a leaky proxy, and a policy that strays far enough can find outputs the RM has never seen and blindly scores high; the KL tethers the search to the region of the distribution the RM has seen; second, to preserve the initial model&rsquo;s language abilities and keep formatting from breaking down. The engineering cost is that a frozen reference model has to stay resident in GPU memory, and every token needs a forward pass through both models.</p>
<p>DAPO removes it entirely (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>β</mi><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\beta=0</annotation></semantics></math></span>, so the reference model doesn&rsquo;t need to be loaded), with the reasoning given in §2.3 of the paper: the KL penalty was originally meant to &ldquo;align, but don&rsquo;t stray too far from the initial model&rdquo;, whereas the goal of long-CoT training is <strong>precisely to depart substantially from the initial model</strong>—the base model cannot write long chains of thought or reflect on itself to begin with; these behaviors depend entirely on RL reshaping the distribution, and staying tethered to the anchor amounts to not letting the ship sail. There is also an unstated enabling condition: RLVR rewards are rule-based judgments, so <strong>there is no RM to hack</strong>, and the anti-hacking half of the KL&rsquo;s motivation simply disappears.</p>
<p>Putting the two premises together reveals an implicit gear shift in DAPO: in the classic setup, drift is held in check by the KL—an <strong>absolute anchor</strong> that forbids straying too far from the initial model; once DAPO drops the KL, only the clip remains—a <strong>relative anchor</strong> that forbids each step from straying too far from the previous one but allows the policy to keep moving farther away. The trust-region role has passed from one anchor to the other. Note also that dropping the KL and raising the upper bound point in the same direction—both loosen the restraints on exploration: the KL penalty pulls the distribution toward the reference model token by token, and only with it gone does the policy dare to venture into new regions. The paper does not say outright that the two were designed to work together—this is my inference; but judged by the goal of &ldquo;preventing collapse&rdquo;, they are indeed fighting in the same trench. Of course, removing the anchor is not free: when training spins out of control, nothing pulls the model back, which is why DAPO relies so heavily on its monitoring dashboard (entropy, length, mean probability), and why it needs substitute stabilizers such as the clip and dynamic sampling.</p>
<h2 id="cross-checking-the-implementations">Cross-Checking the Implementations</h2>
<p>Beyond the mechanism on paper, I checked three implementations, and the details largely line up:</p>
<ul>
<li><strong>rlhf-book</strong>&rsquo;s <code>DAPOLoss</code> in <code>code/policy_gradients/loss.py</code>: decoupled clip (<code>clip_eps_lo=0.2 / clip_eps_hi=0.28</code>, see <code>configs/dapo.yaml</code>), token-level loss normalization (the loss is summed and then divided by the total number of valid tokens across the whole batch, rather than GRPO&rsquo;s original per-sequence mean followed by a mean over sequences), and no KL. The teaching implementation is consistent with the paper.</li>
<li><strong>verl&rsquo;s production implementation</strong> (<code>verl/trainer/ppo/core_algos.py</code>): <code>clip_ratio_low</code> / <code>clip_ratio_high</code> are independent config options that fall back to the symmetric <code>clip_ratio</code> when unset—that is, DAPO&rsquo;s asymmetric clipping is already a first-class citizen, but the default behavior is still classic PPO. The file also contains dual-clip&rsquo;s <code>clip_ratio_c=3.0</code>, which guards against ratio blow-up on negative-advantage samples—a problem in a different direction from the one Clip-Higher solves.</li>
<li><strong>Industry corroboration</strong>: MiniMax&rsquo;s CISPO likewise allows asymmetric clipping bounds; VAPO adopts clip-higher directly. This insight is no longer unique to DAPO; it has become an engineering consensus in long-CoT RL.</li>
</ul>
<p>Let me also mention a fact-checking case: the Chinese-language course hands-on-modern-rl, in its explanation of Clip-Higher, says &ldquo;a lower bound of 0.8 means a low-probability action can be pushed down to 0.008 at most, almost completely suppressed&rdquo;, conflating the suppressive effect of the lower bound with the room to rise under the upper bound, which can easily lead readers to think it is the lower bound that was changed. The paper&rsquo;s actual mechanism is exactly the opposite: what changes is the upper bound; the lower bound stays at 0.2 precisely because loosening it further would crush low-probability tokens. The course states that it was generated with AI assistance and has not yet been fully reviewed; when reading secondhand material, always check back against the original—this very check turned up one such error.</p>
<h2 id="addendum-rlvr-and-grpo-are-not-on-the-same-layer">Addendum: RLVR and GRPO Are Not on the Same Layer</h2>
<p>When reading this post, it is easy to say RLVR and GRPO in one breath, but they are actually two orthogonal axes:</p>
<table>
  <thead>
      <tr>
          <th></th>
          <th>Question it answers</th>
          <th>Which layer</th>
          <th>Other options on the same layer</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><strong>RLVR</strong></td>
          <td>Where does the reward come from?</td>
          <td>Reward source / training paradigm</td>
          <td>Human preferences (RLHF&rsquo;s starting point), learned reward models, verifiable rule-based rewards</td>
      </tr>
      <tr>
          <td><strong>GRPO</strong></td>
          <td>Given the reward, how do you update the model?</td>
          <td>Policy optimization algorithm</td>
          <td>REINFORCE, PPO, DAPO, GSPO</td>
      </tr>
  </tbody>
</table>
<p>The two can be combined freely: RLVR + PPO works perfectly well (VAPO is exactly the value-based route running on verifiable rewards); RM + GRPO holds up too (when GRPO came out of DeepSeekMath it was not tied to rule-based rewards, and within-group normalization works for any scalar reward). RLVR + GRPO became the mainstream recipe because the two bring out the best in each other: RLVR&rsquo;s rule-based rewards are cheap, objective, and robust to reward hacking, but they are a sparse 0/1 signal; GRPO&rsquo;s within-group normalization happens to suit 0/1 well—sample a group for the same problem and the ratio of right to wrong naturally forms a baseline, with no need to train a separate critic to estimate &ldquo;how many points this line of reasoning is worth&rdquo;. Conversely, PPO&rsquo;s critic is hard to train accurately on sparse 0/1 rewards over sequences thousands of tokens long.</p>
<p>Placed back into this post&rsquo;s failure chain, the division of labor between the two is clear as well: RLVR determines <strong>how fragile the signal source is</strong> (0/1 rewards come in only three patterns—all correct, all wrong, or a mix—and the first two give zero gradient outright), while GRPO determines <strong>what the signal depends on</strong> (within-group reward variance, i.e., sampling diversity). All four of DAPO&rsquo;s techniques are patches at GRPO&rsquo;s algorithm layer and leave RLVR&rsquo;s reward source untouched; improvements in the other direction—building more verifiable tasks for RLVR, refining rewards down to the process level (PRM)—are surgery at the reward-paradigm layer, a path orthogonal to the algorithm-layer patches.</p>
<h2 id="my-assessment-and-open-questions">My Assessment and Open Questions</h2>
<p><strong>Assessment</strong>: At its core, Clip-Higher turns the &ldquo;pace of exploration&rdquo; from an implicit default left over from history (ε=0.2, from the continuous-control settings of 2017) into an explicit knob geared to long-CoT reasoning. The 0.28 is not theoretically derived; it is an empirical value, but the diagnostic framework behind it—the asymmetry of ratio clipping on the probability axis—has general explanatory power. From now on, whenever you see &ldquo;training was going fine, then entropy suddenly collapsed&rdquo;, first checking the probability distribution of the tokens held down by the clip should become standard practice.</p>
<p><strong>Open questions</strong>, in order of how much I care about them:</p>
<ol>
<li><strong>Entropy has no target range.</strong> The paper&rsquo;s empirical conclusion is that &ldquo;a slow rise in entropy is beneficial&rdquo;, but it also acknowledges that excessively high entropy corresponds to gibberish and repetition. So should ε_high be adjusted adaptively along with the entropy curve rather than fixed at 0.28? I haven&rsquo;t seen anyone study this systematically.</li>
<li><strong>Order effects in the ablation.</strong> Table 1 is a cumulative chain: Clip-Higher is tested after Overlong Filtering, so its +2 points were measured on a base that already includes earlier changes. What its clean, standalone contribution is remains unknown.</li>
<li><strong>Compatibility with sequence-level clipping.</strong> GSPO lifts the importance sampling ratio to the sequence level (to fix the numerical instability of token-level ratios in MoE models); does the logic of &ldquo;raising the token-level upper bound&rdquo; still hold then? The <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">GSPO post</a>
 unpacks exactly this question.</li>
<li><strong>Robustness under asynchronous training.</strong> In asynchronous RL the ratio distribution drifts further and is more skewed; whether a fixed 0.28 is appropriate across different degrees of off-policyness is not covered by the paper.</li>
</ol>
<h2 id="references">References</h2>
<ul>
<li>DAPO paper: <a href="https://arxiv.org/abs/2503.14476">arXiv:2503.14476</a>
 (primary source for the mechanism, numerical examples, ablation data, and training details)</li>
<li>RLHF Book Chapter 6 and its companion implementation: <a href="https://github.com/natolambert/rlhf-book/blob/main/code/policy_gradients/loss.py"><code>code/policy_gradients/loss.py</code></a>
, <a href="https://github.com/natolambert/rlhf-book/blob/main/code/policy_gradients/configs/dapo.yaml"><code>code/policy_gradients/configs/dapo.yaml</code></a>
 (<a href="https://github.com/natolambert/rlhf-book">natolambert/rlhf-book</a>
)</li>
<li>verl production implementation: <a href="https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/core_algos.py"><code>verl/trainer/ppo/core_algos.py</code></a>
 (clip_ratio_low/high, dual-clip)</li>
<li>Bilibili video: <a href="https://www.bilibili.com/video/BV1LgXbY5EFD">Analysis of the GRPO loss/objective and its biases (DAPO, Dr. GRPO), BV1LgXbY5EFD</a>
 (note: this episode mainly covers dynamic sampling and length bias, and Dr. GRPO&rsquo;s standard-deviation issue; it spends very little time on Clip-Higher)</li>
<li>hands-on-modern-rl <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter18_grpo/deepseek-dapo.md"><code>docs/chapter18_grpo/deepseek-dapo.md</code></a>
 (a Chinese-language explainer; the wording in its Clip-Higher section is muddled, as flagged in the main text)</li>
<li>Notion &ldquo;Common Language&rdquo; notes page (the three rounds of industrial reworking of GRPO: DAPO / Dr.GRPO / GSPO)</li>
</ul>
<hr>
<blockquote>
<p>Update log: 2026-08-26, first version published; the same day, additions based on reader Q&amp;A—the definition and caveats of &ldquo;entropy&rdquo;, the two sources of the advantage in PPO and GRPO and their consequences, how to read the one-sided clipping of the min (value vs. gradient), the precise meaning of the &ldquo;single-round budget&rdquo; and its multiplicative accumulation, the layering of RLVR and GRPO, and a detailed look at the KL penalty and the absolute-anchor/relative-anchor gear shift.</p>
</blockquote>
]]></content:encoded>
    </item>
  </channel>
</rss>
