<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>moe on Njx&#39;Log</title>
    <link>https://njx-njx.github.io/en/tags/moe/</link>
    <description>Recent content in moe on Njx&#39;Log</description>
    <image>
      <title>Njx&#39;Log</title>
      <url>https://njx-njx.github.io/blog-cover.jpg</url>
      <link>https://njx-njx.github.io/blog-cover.jpg</link>
    </image>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate>Tue, 27 Oct 2026 15:14:17 +0800</lastBuildDate>
    <atom:link href="https://njx-njx.github.io/en/tags/moe/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>The Ratio&#39;s Wrong Unit: GSPO Goes Sequence-Level</title>
      <link>https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/</link>
      <pubDate>Wed, 26 Aug 2026 17:40:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/</guid>
      <description>Qwen says GRPO&amp;rsquo;s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.</description>
      <content:encoded><![CDATA[<p>GRPO carries over PPO&rsquo;s token-level importance ratio, which the Qwen team deems a fundamental misuse of importance sampling that leads to irreversible collapse in long-response MoE training. GSPO lifts the ratio, clipping, and optimization all to the sequence level: it clips two orders of magnitude more tokens, yet trains more efficiently. This post unpacks that chain of argument, along with the premises and costs the paper leaves unstated.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/gspo-sequence-level-importance-sampling/">interactive report</a>
 (in Chinese), with drill-down formula breakdowns, a clipping-band comparison, and a dilution-effect calculator.</p>
</blockquote>
<h2 id="the-problem-a-patch-exposed">The Problem a Patch Exposed</h2>
<p>Before mid-2025, training MoE models with GRPO required a dedicated patch.</p>
<p>In the GSPO paper (arXiv:2507.18071), the Qwen team describes the approach they had been using: <strong>Routing Replay</strong>. During sampling, the expert routes activated by the old policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}}</annotation></semantics></math></span> are cached; when computing the importance ratio, these routes are &ldquo;replayed&rdquo; on the new policy <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\pi_\theta</annotation></semantics></math></span>, so that the ratio&rsquo;s numerator and denominator run through the same activated network. The patch was needed because of one number they observed: in the 48-layer Qwen3-30B-A3B, after each gradient update, roughly 10% of the experts activated for the same rollout sample under the new policy differ from those under the old policy. Once the routing changes, the per-token probabilities change with it, the token-level importance ratios fluctuate violently, and GRPO simply fails to converge.</p>
<p>The patch works, but its costs are real: extra GPU memory and communication overhead, and forcing the new policy onto the old routes amounts to capping the model&rsquo;s effective capacity. What is more troubling is something else — even with the patch, GRPO on long-response tasks with large models still suffers <strong>irreversible model collapse</strong>. The paper&rsquo;s wording is strong: once collapse occurs, rolling back to a checkpoint, carefully tuning the clipping range, extending the generation length, switching RL queries — none of it can bring the model back.</p>
<p>The token-level importance ratio has been standard since the PPO era, in use for eight years. Why has it suddenly turned pathological in RL for LLMs? GSPO&rsquo;s answer: it did not suddenly fall ill; it was used in the wrong place from the start — this is a misuse of importance sampling, and in the era of small models, short responses, and near-on-policy training, the error was simply masked.</p>
<h2 id="why-importance-sampling-fails">Why Importance Sampling &ldquo;Fails&rdquo;</h2>
<p>Let me first write down importance sampling (IS) itself. We want to use samples from a behavior distribution <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mrow><mi>b</mi><mi>e</mi><mi>h</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\pi_{beh}</annotation></semantics></math></span> to estimate the expectation, under a target distribution <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mrow><mi>t</mi><mi>a</mi><mi>r</mi></mrow></msub></mrow><annotation encoding="application/x-tex">\pi_{tar}</annotation></semantics></math></span>, of a function <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi></mrow><annotation encoding="application/x-tex">f</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="double-struck">E</mi><mrow><mi>z</mi><mo>∼</mo><msub><mi>π</mi><mrow><mi>t</mi><mi>a</mi><mi>r</mi></mrow></msub></mrow></msub><mo stretchy="false">[</mo><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>=</mo><msub><mi mathvariant="double-struck">E</mi><mrow><mi>z</mi><mo>∼</mo><msub><mi>π</mi><mrow><mi>b</mi><mi>e</mi><mi>h</mi></mrow></msub></mrow></msub><mrow><mo fence="true">[</mo><mfrac><mrow><msub><mi>π</mi><mrow><mi>t</mi><mi>a</mi><mi>r</mi></mrow></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><mrow><msub><mi>π</mi><mrow><mi>b</mi><mi>e</mi><mi>h</mi></mrow></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow></mfrac><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\mathbb{E}_{z\sim\pi_{tar}}[f(z)] = \mathbb{E}_{z\sim\pi_{beh}}\left[\frac{\pi_{tar}(z)}{\pi_{beh}(z)} f(z)\right]</annotation></semantics></math></span><p>This identity does not hold by virtue of the ratio itself; it relies on <strong>averaging over many samples</strong>: only when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>≫</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">N \gg 1</annotation></semantics></math></span> do the fluctuations of the random ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>w</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi>π</mi><mrow><mi>t</mi><mi>a</mi><mi>r</mi></mrow></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo><mi mathvariant="normal">/</mi><msub><mi>π</mi><mrow><mi>b</mi><mi>e</mi><mi>h</mi></mrow></msub><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">w(z)=\pi_{tar}(z)/\pi_{beh}(z)</annotation></semantics></math></span> average out, so that the estimate converges to the true value. Multiplying a single sample by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>w</mi></mrow><annotation encoding="application/x-tex">w</annotation></semantics></math></span> provides no correction whatsoever; it merely multiplies <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>f</mi><mo stretchy="false">(</mo><mi>z</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">f(z)</annotation></semantics></math></span> by a high-variance random number.</p>
<p>Now look at what GRPO does. It computes a ratio for every token:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">w_{i,t}(\theta)=\frac{\pi_\theta(y_{i,t}|x,y_{i,&lt;t})}{\pi_{\theta_{old}}(y_{i,t}|x,y_{i,&lt;t})}</annotation></semantics></math></span><p>Here <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">y_{i,t}</annotation></semantics></math></span> is the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>i</mi></mrow><annotation encoding="application/x-tex">i</annotation></semantics></math></span>-th response&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span>-th token, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">y_{i,&lt;t}</annotation></semantics></math></span> is the context preceding it. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><mo>⋅</mo><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi_{\theta_{old}}(\cdot|x,y_{i,&lt;t})</annotation></semantics></math></span> is the next-token distribution at step <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span>, and <strong>only one sample</strong> is drawn from it: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">y_{i,t}</annotation></semantics></math></span>. In other words, every ratio uses a single sample to &ldquo;correct&rdquo; a single-step distribution — by the criterion above, this does not constitute an IS correction at all; it merely injects a per-token random weight into the gradient. The longer the sequence, the more random weights accumulate; the larger and sparser the model (MoE), the higher the variance of each individual weight. Clipping then zeroes out the gradients of tokens that fall outside the range, and the noise is amplified into &ldquo;randomly wiping out the learning signal of a subset of tokens&rdquo;. Chapter 6 of the RLHF Book gives two concrete descriptions of the failure as it plays out: a single token with a large ratio can dominate the entire update; and tokens within the same response are clipped independently of one another, shredding the learning signal.</p>
<p>The GSPO paper distills the crux into a single sentence: <strong>the unit of the optimization objective should match the unit of the reward</strong>. In RLVR, the reward is assigned to the entire sequence (1 point for a correct answer, 0 for a wrong one), whereas GRPO&rsquo;s correction and clipping both happen at the token level. With the units mismatched, the more &ldquo;conscientious&rdquo; the correction, the more precisely targeted the damage.</p>
<p>To be honest, I should add one point here: GRPO&rsquo;s formulation is not without theoretical grounding. Treating each token as one action step and decomposing the IS ratio of the whole trajectory into per-step ratios is the standard derivation path in RL. The problem lies at the implementation level, where the per-token ratios are used one by one as independent correction weights and clipped one by one — an approximation that stops being harmless under large vocabularies, long trajectories, and single samples. This is the GSPO paper&rsquo;s critical angle, not an axiom; readers are free to reach their own judgment.</p>
<h2 id="gspo-moving-ratio-clipping-and-optimization-to-the-sequence-level">GSPO: Moving Ratio, Clipping, and Optimization to the Sequence Level</h2>
<p>The GSPO objective:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="script">J</mi><mrow><mi>G</mi><mi>S</mi><mi>P</mi><mi>O</mi></mrow></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msub><mi mathvariant="double-struck">E</mi><mrow><mi>x</mi><mo>∼</mo><mi mathvariant="script">D</mi><mo separator="true">,</mo><mo stretchy="false">{</mo><msub><mi>y</mi><mi>i</mi></msub><msubsup><mo stretchy="false">}</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></msubsup><mo>∼</mo><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub></mrow></msub><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><mi>min</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo separator="true">,</mo><mtext> </mtext><mrow><mi mathvariant="normal">c</mi><mi mathvariant="normal">l</mi><mi mathvariant="normal">i</mi><mi mathvariant="normal">p</mi></mrow><mo stretchy="false">(</mo><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo separator="true">,</mo><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo fence="true">)</mo></mrow><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\mathcal{J}_{GSPO}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\{y_i\}_{i=1}^G\sim\pi_{\theta_{old}}}\left[\frac{1}{G}\sum_{i=1}^G \min\left(s_i(\theta)\hat{A}_i,\ \mathrm{clip}(s_i(\theta),1-\varepsilon,1+\varepsilon)\hat{A}_i\right)\right]</annotation></semantics></math></span><p>The difference from GRPO is visible at a glance: the per-token sum <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><msub><mo>∑</mo><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">\frac{1}{|y_i|}\sum_t</annotation></semantics></math></span> outside the min-clip is gone, and each response as a whole occupies just one term in the objective. The advantage is still the group-normalized <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">\hat{A}_i</annotation></semantics></math></span> (standardized with the mean and standard deviation of the rewards within the group), shared by the entire response.</p>
<p>The core is the definition of the sequence-level ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><msup><mrow><mo fence="true">(</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo stretchy="false">)</mo></mrow><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo stretchy="false">)</mo></mrow></mfrac><mo fence="true">)</mo></mrow><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac></msup><mo>=</mo><mi>exp</mi><mo>⁡</mo><mrow><mo fence="true">(</mo><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><mi>log</mi><mo>⁡</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><mrow><msub><mi>π</mi><msub><mi>θ</mi><mrow><mi>o</mi><mi>l</mi><mi>d</mi></mrow></msub></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow></mfrac><mo fence="true">)</mo></mrow></mrow><annotation encoding="application/x-tex">s_i(\theta)=\left(\frac{\pi_\theta(y_i|x)}{\pi_{\theta_{old}}(y_i|x)}\right)^{\frac{1}{|y_i|}}=\exp\left(\frac{1}{|y_i|}\sum_{t=1}^{|y_i|}\log\frac{\pi_\theta(y_{i,t}|x,y_{i,&lt;t})}{\pi_{\theta_{old}}(y_{i,t}|x,y_{i,&lt;t})}\right)</annotation></semantics></math></span><p>This is the geometric mean of the whole-sequence likelihood ratio, equivalent to taking the arithmetic mean of the per-token log-ratios and then exponentiating. The length normalization <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> cannot be dropped: without it, thousands of ratios get multiplied together, likelihood changes in just a few tokens can make the overall ratio swing violently, and responses of different lengths would need different clipping ranges — there would be no way to set <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> at all. After normalization, the numerical range of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> is decoupled from sequence length, and a single <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> covers everything. (The hands-on-modern-rl textbook omits this <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y_i|</annotation></semantics></math></span> from its formula when introducing GSPO in Section 16.4; go by Equation 7 in the paper — leave it out and the whole chain of argument breaks.)</p>
<p>The meaning of clipping changes accordingly. In GRPO, clipping zeroes out the gradient of an individual token — the signal is chopped up, but the sample remains; in GSPO, clipping removes an entire response from the gradient estimate — a sample is either used in full or not at all. This aligns, in granularity, with &ldquo;the reward is given to the whole sequence&rdquo;.</p>
<h2 id="the-gradient-view-unequal-vs-equal-weighting">The Gradient View: Unequal vs. Equal Weighting</h2>
<p>Expand the gradients of the two objectives (Equations 8–12 in the paper), and the fundamental difference comes down to one thing: what multiplies each token&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub></mrow><annotation encoding="application/x-tex">\nabla_\theta\log\pi_\theta</annotation></semantics></math></span>.</p>
<p>In GRPO&rsquo;s gradient, token <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span>&rsquo;s <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∇</mi><mi>log</mi><mo>⁡</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">\nabla\log\pi</annotation></semantics></math></span> is multiplied by <strong>its own ratio</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">w_{i,t}</annotation></semantics></math></span>. After the min-clip, these per-token weights lie in <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">(</mo><mn>0</mn><mo separator="true">,</mo><mn>1</mn><mo>+</mo><mi>ε</mi><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">(0, 1+\varepsilon]</annotation></semantics></math></span> (when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A}_i&gt;0</annotation></semantics></math></span>) or <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">[</mo><mn>1</mn><mo>−</mo><mi>ε</mi><mo separator="true">,</mo><mo>+</mo><mi mathvariant="normal">∞</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">[1-\varepsilon, +\infty)</annotation></semantics></math></span> (when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>&lt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A}_i&lt;0</annotation></semantics></math></span>) — within the same response, the gradient weights that different tokens receive can differ by orders of magnitude, and as they accumulate over training steps, the consequences are hard to predict.</p>
<p>In GSPO&rsquo;s gradient, all tokens of the same response share <strong>the same scalar weight</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><msub><mi mathvariant="script">J</mi><mrow><mi>G</mi><mi>S</mi><mi>P</mi><mi>O</mi></mrow></msub><mo>=</mo><mi mathvariant="double-struck">E</mi><mrow><mo fence="true">[</mo><mfrac><mn>1</mn><mi>G</mi></mfrac><munderover><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>G</mi></munderover><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><msub><mover accent="true"><mi>A</mi><mo>^</mo></mover><mi>i</mi></msub><mo>⋅</mo><mfrac><mn>1</mn><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></mfrac><munderover><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mrow><mi mathvariant="normal">∣</mi><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi></mrow></munderover><msub><mi mathvariant="normal">∇</mi><mi>θ</mi></msub><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mo fence="true">]</mo></mrow></mrow><annotation encoding="application/x-tex">\nabla_\theta\mathcal{J}_{GSPO} = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^G s_i(\theta)\hat{A}_i \cdot \frac{1}{|y_i|}\sum_{t=1}^{|y_i|} \nabla_\theta\log\pi_\theta(y_{i,t}|x,y_{i,&lt;t})\right]</annotation></semantics></math></span><p>The tokens&rsquo; contributions to the update differ only in the magnitude of their own <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">∇</mi><mi>log</mi><mo>⁡</mo><mi>π</mi></mrow><annotation encoding="application/x-tex">\nabla\log\pi</annotation></semantics></math></span>; there is no longer any per-token random weight modulating them. The paper describes this as eliminating the destabilizing factor in GRPO.</p>
<h2 id="evidence-three-experimental-facts">Evidence: Three Experimental Facts</h2>
<p>The paper&rsquo;s experimental setup is itself worth noting: a cold-start model (obtained by SFT from Qwen3-30B-A3B-Base), with each rollout batch split into 4 mini-batches for gradient updates — a standard off-policy setting, and precisely the setting in which, as the previous post on Clip-Higher put it, &ldquo;clipping actually starts to do its job&rdquo; (only then does the ratio deviate from 1). The GRPO baseline&rsquo;s clipping range of 0.2/0.27 was carefully tuned; GSPO uses 3e-4 / 4e-4.</p>
<p>Fact one: <strong>the three-orders-of-magnitude gap in ε is not a typo; it is an inevitable consequence of averaging</strong>. A single token-level ratio can easily shoot up to 1.2; the sequence-level ratio is the exp of the average of the per-token log-ratios, and averaging cancels out positive and negative fluctuations, so <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> naturally stays close to 1, fluctuating only slightly. This can be worked out in detail (what follows is my own estimate; the paper does not report these statistics): suppose that after one mini-batch update, each token&rsquo;s log-ratio drift has, on average, a standard deviation of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math></span> (say, 5%), the response length is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span>, and the drifts of different tokens are approximately independent; then the standard deviation of the averaged drift shrinks to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mi mathvariant="normal">/</mi><msqrt><mi>L</mi></msqrt></mrow><annotation encoding="application/x-tex">\sigma/\sqrt{L}</annotation></semantics></math></span> — for <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo>=</mo><mn>2000</mn></mrow><annotation encoding="application/x-tex">L=2000</annotation></semantics></math></span>, a reduction to about <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mn>45</mn></mrow><annotation encoding="application/x-tex">1/45</annotation></semantics></math></span>. Give such a nearly motionless ratio a clipping band of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>±</mo><mn>0.2</mn></mrow><annotation encoding="application/x-tex">\pm 0.2</annotation></semantics></math></span> and it will never touch the edges; clipping might as well not exist. So GSPO has to tighten the band to a few parts in ten thousand: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.2</mn><mi mathvariant="normal">/</mi><mn>3</mn><mo>×</mo><msup><mn>10</mn><mrow><mo>−</mo><mn>4</mn></mrow></msup><mo>≈</mo><mn>667</mn></mrow><annotation encoding="application/x-tex">0.2 / 3\times10^{-4} \approx 667</annotation></semantics></math></span> — that is where the three-orders-of-magnitude narrowing comes from. Put another way: a tolerance band of 3e-4 corresponds to allowing an average log drift of only about 0.03% per token. I should also be upfront about how sound this assumption is: in real training, per-token drifts are not independent (a single update raises or lowers the probabilities of similar contexts en masse), so the actual reduction factor falls short of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msqrt><mi>L</mi></msqrt></mrow><annotation encoding="application/x-tex">\sqrt{L}</annotation></semantics></math></span> — but this only makes the conclusion of Fact two below stronger.</p>
<p>Fact two: <strong>a hundred times more tokens get clipped, yet efficiency is higher</strong>. The band is a thousand times narrower, yet a hundred times more tokens are clipped — this sounds contradictory, but once you multiply two factors together, it isn&rsquo;t (the estimates below are, as above, my own). First factor: relative to their respective fluctuation scales, the band is actually narrower. The fluctuation shrinks by only about 45×, while the band shrinks by 667×; using each ratio&rsquo;s own fluctuation as the yardstick, GSPO&rsquo;s band is about 15× narrower than GRPO&rsquo;s (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msqrt><mi>L</mi></msqrt><mi mathvariant="normal">/</mi><mn>667</mn></mrow><annotation encoding="application/x-tex">\sqrt{L}/667</annotation></semantics></math></span>), so <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> goes out of bounds far more easily than <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>w</mi><mi>t</mi></msub></mrow><annotation encoding="application/x-tex">w_t</annotation></semantics></math></span> does. Plugging in concrete numbers (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mo>=</mo><mn>5</mn><mi mathvariant="normal">%</mi></mrow><annotation encoding="application/x-tex">\sigma=5\%</annotation></semantics></math></span>, approximately Gaussian drift): GRPO&rsquo;s 0.2 corresponds to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>4</mn><mi>σ</mi></mrow><annotation encoding="application/x-tex">4\sigma</annotation></semantics></math></span>, so the probability of a single token going out of bounds is a few in ten thousand; GSPO&rsquo;s 3e-4 corresponds to only about <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.27</mn><mi>σ</mi></mrow><annotation encoding="application/x-tex">0.27\sigma</annotation></semantics></math></span> (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0.0003</mn><mi mathvariant="normal">/</mi><mo stretchy="false">(</mo><mn>0.05</mn><mi mathvariant="normal">/</mi><mn>45</mn><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">0.0003/(0.05/45)</annotation></semantics></math></span>), so each response has a better-than-even chance of being thrown out in its entirety. Second factor: going out of bounds is charged to the whole response. In GRPO, a token that goes out of bounds discards only itself; in GSPO, a single out-of-bounds event removes the entire span of several thousand tokens. Multiply the two factors, and the total number of clipped tokens comes to roughly a hundred times GRPO&rsquo;s — the gap in clipping fractions measured in Figure 2 of the paper is indeed about two orders of magnitude, so the magnitudes line up. The truly counterintuitive part is the second half of the claim: fewer tokens are used for training, yet training efficiency and benchmark performance are better (winning on all three curves: AIME'24 / LiveCodeBench / CodeForces). The paper&rsquo;s interpretation: GRPO&rsquo;s token-level gradient estimates are noisy to begin with, and the portion that happens to survive is no more useful than the portion that gets clipped; GSPO&rsquo;s sequence-level filtering provides a more reliable learning signal — fewer but clean beats more but dirty.</p>
<p>Fact three: <strong>MoE no longer needs Routing Replay</strong>. The sequence likelihood <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>i</mi></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\pi_\theta(y_i|x)</annotation></semantics></math></span> is insensitive to routing jitter — when 10% of the experts change, individual token probabilities may shift noticeably, but the likelihood of the whole sequence (a reflection of the model&rsquo;s language-modeling capability) does not change drastically. GSPO cares only about the latter, so simply computing the ratio as usual is enough for it to converge; the patch&rsquo;s memory and communication overhead is saved entirely, and the model is no longer bound to the old routes.</p>
<p>There is also an infrastructure dividend on the side (§5.4 of the paper): in practice, the training engine (Megatron) and the inference engines (SGLang, vLLM) differ in numerical precision, so GRPO has to recompute the old policy&rsquo;s likelihoods with the training engine before it can safely use them; sequence-level likelihoods are far more tolerant of precision discrepancies, so GSPO can optimize directly with the likelihoods returned by the inference engine and skip the recomputation. For partial rollouts, multi-turn RL, and architectures that decouple training from inference, this is a tangible throughput dividend.</p>
<h2 id="appendix-how-the-3e-44e-4-clipping-range-was-set">Appendix: How the 3e-4/4e-4 Clipping Range Was Set</h2>
<p>At the level of the paper: it isn&rsquo;t stated. The GSPO paper gives only the result (left bound 3e-4, right bound 4e-4), with no selection procedure and no ablation on the clipping range. Why this value rather than 1e-4 or 1e-3 is impossible for outsiders to know; a reasonable guess is that the authors found it by trial and error on their own setup (cold-start Qwen3-30B-A3B, each batch split into 4 mini-batches), but that is only a guess.</p>
<p>At the level of principle (what follows is my inference, not the paper&rsquo;s content): the clipping range&rsquo;s job is to let normal fluctuations through and stop abnormal ones, so it should be set at a small multiple of the typical fluctuation magnitude of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mi>ε</mi><mo>≈</mo><mi>k</mi><mo>⋅</mo><mfrac><mi>σ</mi><msqrt><mi>L</mi></msqrt></mfrac></mrow><annotation encoding="application/x-tex">\varepsilon \approx k \cdot \frac{\sigma}{\sqrt{L}}</annotation></semantics></math></span><p>Here <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math></span> is the magnitude of the per-token log-ratio drift after one update, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span> is the response length, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span> is the multiple of the fluctuation you want to let through. Plugging in the illustrative numbers from earlier (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mo>=</mo><mn>5</mn><mi mathvariant="normal">%</mi></mrow><annotation encoding="application/x-tex">\sigma=5\%</annotation></semantics></math></span>, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi><mo>=</mo><mn>2000</mn></mrow><annotation encoding="application/x-tex">L=2000</annotation></semantics></math></span>) gives <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mi mathvariant="normal">/</mi><msqrt><mi>L</mi></msqrt><mo>≈</mo><mn>0.0011</mn></mrow><annotation encoding="application/x-tex">\sigma/\sqrt{L} \approx 0.0011</annotation></semantics></math></span>, so the paper&rsquo;s 3e-4 corresponds to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi><mo>≈</mo><mn>0.27</mn></mrow><annotation encoding="application/x-tex">k \approx 0.27</annotation></semantics></math></span> — letting through only about a quarter of the typical fluctuation, a very strict filter, which squares with the measured &ldquo;tens of percent of tokens clipped&rdquo;.</p>
<p>This framework also shows that 3e-4 cannot be copied as-is; it is coupled to three things: the degree of off-policyness (the staler the data, the larger <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi></mrow><annotation encoding="application/x-tex">\sigma</annotation></semantics></math></span>, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> must be loosened accordingly, or else every sample gets clipped away); response length (the longer <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>L</mi></mrow><annotation encoding="application/x-tex">L</annotation></semantics></math></span>, the smaller the fluctuation after averaging, so the same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> becomes stricter — suitable values for 32k and 2k differ by about 4×); and the desired clipping strength (there is no public data on the optimal clipping fraction, so a reproduction has to run its own sweep). Change the configuration, and this number has to be set again.</p>
<h2 id="my-three-reservations">My Three Reservations</h2>
<p>First, <strong>the &ldquo;single-sample&rdquo; critique applies equally to GSPO itself</strong>. <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> is also a ratio computed from a single sample <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>y</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">y_i</annotation></semantics></math></span>, likewise without any <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mo>≫</mo><mn>1</mn></mrow><annotation encoding="application/x-tex">N \gg 1</annotation></semantics></math></span> averaging. So GSPO&rsquo;s improvement should not be understood as &ldquo;restoring the theoretical correctness of importance sampling&rdquo; — it does no such thing; the paper&rsquo;s IS argument is intuitive, not formal. What it really does is bring three things together: unit alignment (what gets clipped is an entire &ldquo;bad sample&rdquo; rather than random tokens), variance normalization (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>1</mn><mi mathvariant="normal">/</mi><mi mathvariant="normal">∣</mi><mi>y</mi><mi mathvariant="normal">∣</mi></mrow><annotation encoding="application/x-tex">1/|y|</annotation></semantics></math></span> squeezes the numerical range onto a common scale), and equal-weight gradients (eliminating the accumulation of unequal per-token weights). All three are engineering arguments — solid, but not theorems.</p>
<p>Second, <strong>the geometric mean is a diluter, and that cuts both ways</strong>. In a 10,000-token CoT, if the probability of one critical token drifts by 50%, its contribution to <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> is only <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>e</mi><mrow><mi>log</mi><mo>⁡</mo><mn>1.5</mn><mi mathvariant="normal">/</mi><mn>10000</mn></mrow></msup><mo>≈</mo><mn>1.00004</mn></mrow><annotation encoding="application/x-tex">e^{\log 1.5 / 10000} \approx 1.00004</annotation></semantics></math></span> — nearly invisible. Against precision noise between the training and inference engines, dilution is a feature (this is exactly where the infrastructure dividend comes from); for detecting &ldquo;a critical token altered by a policy update&rdquo;, dilution is a loophole. GSPO works properly precisely because it is paired with an extremely narrow clipping band on the order of three parts in ten thousand — making trust-region judgments based on average drift rather than single-point drift. Whether this combination goes blind to certain failure modes on ultra-long CoTs (for example, the policy deviating sharply on only a short stretch of reasoning, which gets let through after averaging) is something the paper does not answer.</p>
<p>Third, <strong>sequence-level clipping gives up on making use of &ldquo;partially correct&rdquo; samples</strong>. If a response gets 90% of its reasoning right but miscalculates in the final step, GSPO either uses it whole or throws it out whole. The paper&rsquo;s way out is the GSPO-token variant: using stop-gradient, it combines the sequence-level weight <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span> (taking only its value, without backpropagating gradients through it) with the per-token term <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>π</mi><mi>θ</mi></msub><mi mathvariant="normal">/</mi><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\pi_\theta / \mathrm{sg}[\pi_\theta]</annotation></semantics></math></span>; the resulting objective, clipping condition, and theoretical gradient are fully equivalent to GSPO&rsquo;s, but advantages can be customized per token — an opening left for multi-turn RL and fine-grained rewards. At the other end of this line is VAPO: bring the value model back outright to do token-level credit assignment. Sequence-level vs. token-level is not a dispute over right and wrong, but two industrial answers to the old question of &ldquo;credit-assignment granularity&rdquo;.</p>
<h2 id="how-it-relates-to-clip-higher-and-a-detail-no-one-has-pointed-out">How It Relates to Clip-Higher, and a Detail No One Has Pointed Out</h2>
<p>The previous post covered DAPO&rsquo;s Clip-Higher: the pathology is the same — a structural flaw of token-level clipping under off-policy training. DAPO&rsquo;s remedy is to adjust the bounds (raising the upper bound from 0.2 to 0.28 to cut exploratory tokens some slack); GSPO&rsquo;s remedy is to change the unit (the whole response shares a single ratio). The two are not mutually exclusive — Qwen&rsquo;s practice is precisely GSPO stacked with a Dr.GRPO-style bias correction that removes the normalization.</p>
<p>There is one more detail worth pointing out: the GSPO paper&rsquo;s clipping range of 3e-4 / 4e-4 is itself <strong>asymmetric</strong> — the right bound is wider than the left, leaving 33% more room in the reinforcing direction (the ratio breaking upward when <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>A</mi><mo>^</mo></mover><mo>&gt;</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">\hat{A}&gt;0</annotation></semantics></math></span>). Isn&rsquo;t that just a sequence-level version of Clip-Higher? The paper does not discuss where this asymmetry comes from, but its direction is consistent with DAPO&rsquo;s logic: let good samples go a little further. Two teams independently prescribed remedies pointing in the same direction for the same pathology — that says more than any single paper&rsquo;s ablation could.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>The method for choosing <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi></mrow><annotation encoding="application/x-tex">\varepsilon</annotation></semantics></math></span> has itself not been disclosed: 3e-4/4e-4 comes with results but no ablation, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>ε</mi><mo>≈</mo><mi>k</mi><mo>⋅</mo><mi>σ</mi><mi mathvariant="normal">/</mi><msqrt><mi>L</mi></msqrt></mrow><annotation encoding="application/x-tex">\varepsilon \approx k \cdot \sigma/\sqrt{L}</annotation></semantics></math></span> is only this post&rsquo;s inferred framework; the optimal range of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span>, and the effect of combining it with clip-higher-style asymmetric bounds, all call for public reproduction;</li>
<li>Whether the dilution effect of the geometric mean systematically lets &ldquo;abrupt local shifts&rdquo; slip through on ultra-long CoTs (beyond 32k tokens) needs to be verified with purpose-built experiments;</li>
<li>The actual effectiveness of GSPO-token paired with turn-level rewards in multi-turn agentic RL is, for now, backed only by Qwen&rsquo;s internal data.</li>
</ul>
<hr>
<blockquote>
<p><strong>Update log (2026-08-29)</strong>: Expanded based on feedback from three rounds of reader Q&amp;A — the &ldquo;Evidence&rdquo; section now includes the full derivation of the clipping-band arithmetic (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>σ</mi><mi mathvariant="normal">/</mi><msqrt><mi>L</mi></msqrt></mrow><annotation encoding="application/x-tex">\sigma/\sqrt{L}</annotation></semantics></math></span> variance reduction, the 667× narrowing, the band being ~15× narrower in relative terms, and the out-of-bounds probability calculation); a new section, &ldquo;Appendix: How the 3e-4/4e-4 Clipping Range Was Set&rdquo;, was added (the paper does not disclose its selection method; an inferred framework and three couplings); the first open question was rewritten accordingly. All derivations are labeled estimates, not content from the paper.</p>
</blockquote>
<h2 id="references">References</h2>
<ul>
<li>GSPO paper: <a href="https://arxiv.org/abs/2507.18071">arXiv:2507.18071v2</a>
; the equation numbers and experimental data in this post all come from its v2</li>
<li>RLHF Book, Chapter 6 (importance sampling background; GSPO/CISPO subsections): <a href="https://rlhfbook.com/">rlhfbook.com</a>
; <a href="https://github.com/natolambert/rlhf-book/blob/main/book/chapters/06-policy-gradients.md"><code>book/chapters/06-policy-gradients.md</code></a>
</li>
<li>hands-on-modern-rl, Section 16.4 (comparison of the GRPO family of improvements; note that its GSPO formula omits the length normalization): <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter18_grpo/grpo-family.md"><code>docs/chapter18_grpo/grpo-family.md</code></a>
</li>
<li>Previous post: <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
</li>
</ul>
]]></content:encoded>
    </item>
  </channel>
</rss>
