[{"anchor":"","content":"The term RSI (recursive self-improvement) became hot again in 2025-2026, but the things it refers to often get lumped together. Starting from a practitioner’s dichotomy (change the harness but not the weights vs. change the weights directly), this post breaks “self-improvement” into three levels—the data loop, the reward loop, and the system loop—which differ in their preconditions, how far they have actually paid off, and their failure modes.\nThis post has a companion interactive report (in Chinese), with an interactive diagram of the three nested loops, a three-level exam matrix, and a scissors-gap diagram.","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"“The Model Trains Itself”: RSI and Three Loops","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"a-conflated-term","content":"“RSI” (Recursive Self-Improvement) became hot again in 2025–2026, but what people mean by it varies widely: some mean an iterative pipeline that filters rollouts into data and retrains on it, some mean a model rewriting its own training process, and some mean scaling up test-time compute. The term’s degree of confusion is directly proportional to the allure of what it promises.\nLet me start with a practitioner’s framework. In a conversation, Ding Duo, head of post-training at Perplexity, split RSI and continual learning into two routes: one leaves the model parameters unchanged and optimizes the harness, the loop, and the solving process; the other updates the model weights directly. His judgment is that the former produces results faster, while the latter may bring deeper changes in capability; and the two routes share three hard problems—controlling cost, avoiding parameter contamination, and proving that what is continually learned is transferable capability. This post will keep coming back to these three problems.\nAnother common confusion is around the term test-time. Classic TTT (Test-Time Training, Sun et al. 2020) refers to taking gradient updates on the current input at inference time, whereas the “test-time scaling” popularized by s1 refers to lengthening the reasoning chain and adding budget, without touching the weights. When people in an LLM context talk about a “new test-time paradigm”, always ask which one they mean—the former actually changes the model; the latter just thinks more. What Ding Duo calls “optimizing the harness/loop” is in fact a third path: leave the gradients alone and change the scaffolding.","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"A Conflated Term","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"level-one-the-data-loopmost-mature-and-least-like-rsi","content":"The first level is data circulating through the loop: model rollout → filter successful trajectories by verifiable outcomes → mix them into the next round of SFT/RL training → the model gets stronger → roll out again. LoopTool (arXiv:2511.09148) has exactly this shape (evaluation uncovers weaknesses → targeted data synthesis → retraining); a more complete industrial example is the WebSailor series (arXiv:2507.02592; the V2 version, 2509.13305, explicitly puts synthetic data + scalable RL in its title): use a strong model to generate browsing trajectories, train with outcome rewards, filter with self-critique, then automatically synthesize harder training tasks. DeepResearcher (2504.03160) reports a phenomenon worth remembering: in this kind of closed-loop training, behaviors such as pre-search planning, information layering, and cross-validation are not explicitly encouraged by any reward term; they are by-products of RL optimization.\nThis level is already industrialized, but strictly speaking it is not “self”-improvement—the choices in the loop are made by a filter designed by engineers, and the model is just a data source. Its ceiling is also clear: the filtering criteria themselves are hackable (inefficient trajectories that “happened to succeed” get kept), and the narrower the data distribution, the faster the loop turns, and the faster the model converges itself toward whatever the current verifier likes.","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"Level One: The Data Loop—Most Mature, and Least Like RSI","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"level-two-the-reward-looprlsvr-aims-to-extend-the-self-verification-dividend-to-open-ended-tasks","content":"The second level is reward circulating through the loop. RLVR’s success over the past two years rests on “math and code have deterministic verifiers”; open-ended tasks (writing, research, dialogue) have no verifier and can only rely on preference models or LLM judges, which brings a triple tax: evaluation bias, the judge-capability bottleneck, and inference cost. RLSVR (arXiv:2607.23802) borrows the concept of a “pretext task” from self-supervised learning: apply a task transformation to the open-ended task to turn it into a self-verifiable form, so that the reward once again comes from the task itself rather than from an external judge.\nThe conditions for this idea to hold are worth laying out: it requires that the task admit some fidelity-preserving transformation (getting the transformed task right ≈ doing the original task well), and that the verifier be deterministic in the transformed space. The “task transformation induces self-verifiable rewards” in the paper’s title refers to exactly this inducing step. If it holds, open-ended tasks can also tap into the dividends of RLVR’s industrialization; if it doesn’t—if the transformation loses too much—the self-verifiable reward is optimizing a proxy metric that has come apart from the original task.\nWhat this means for the loops: the reward loop is the most critical of the three levels for RSI. The data loop’s filter and the system loop’s evaluator both have to answer “what counts as getting better”, and self-verifiable reward is currently the only answer that does not depend on a continuous supply from humans.","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"Level Two: The Reward Loop—RLSVR Aims to Extend the Self-Verification Dividend to Open-Ended Tasks","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"level-three-the-system-looprsi-proper-stuck-on-oversight","content":"Only the third level is RSI proper: the model (or what it produces) improves the things used to improve it—better assistants, better data, better evaluations—forming a recursion. Leike et al.’s 2018 research agenda for reward modeling (arXiv:1811.07871) already has a section explicitly discussing recursive reward modeling: use AI to help humans evaluate AI outputs, so that evaluation capability grows along with whatever is being evaluated.\nThe prerequisite for this level is scalable oversight (2211.03540, cited in Chapter 30 of the hands-on-modern-rl textbook as a core direction in alignment): once model capability exceeds human evaluation capability, who verifies that an improvement is real? Every round of the loop makes selections, and once the selection criteria are hackable, the faster the loop turns, the faster it collapses—a small-scale version is already visible in the data loop (metric inflation), and the system loop amplifies it to the entire pipeline. This is also why “proving that what is continually learned is transferable capability” is on Ding Duo’s list of hard problems: improvement inside the loop is often “getting stronger on this loop’s evaluations”, which does not necessarily hold outside the loop.","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"Level Three: The System Loop—RSI Proper, Stuck on Oversight","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"the-three-levels-share-one-exam","content":"Putting the three levels together, each of Ding Duo’s three hard problems maps onto concrete mechanisms:\nCost: the data loop’s cost is rollout and filtering (every iteration samples in full; we did the rollout = prompt × G math earlier in the series); the reward loop’s cost is constructing and maintaining the task transformations and verifiers; the system loop’s cost is basically impossible to estimate for now—because no instance has yet run the full pipeline end to end. Parameter contamination: only level three is involved. Continual gradient updates let new knowledge interfere with old capabilities, and the contamination accumulates with the number of iterations—this is one reason the “don’t change the weights” route is currently underrated: optimizing the harness and loop doesn’t contaminate parameters, and the gains are immediate; engineering work like DeepSeek V3.2 (self-added verifiers, switching the sampler to mHC) is this route paying off. Transferability: none of the three levels has a good evaluation protocol. Evaluation inside a loop is contaminated by the loop itself (the training target and the evaluation target share the same source), and cross-loop evaluation is too expensive.","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"The Three Levels Share One Exam","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"my-three-takes","content":"First, the “don’t change the weights” route is currently the main way RSI actually pays off, and theorists underrate it. Scaffold optimization (tools, context engineering, verifier internalization, sampling procedures) doesn’t change parameters, doesn’t contaminate weights, and can be rolled back, and at the current stage its return per unit of compute is higher than that of changing the weights directly. Expelling it from the ranks of “self-improvement” is a misallocation of resources caused by terminological purism.\nSecond, the three levels are bottlenecked in different places. The data loop is stuck on the hackability of its filter; the reward loop on the fidelity of the task transformation; the system loop on oversight (without a solution to scalable oversight, recursion can only be recursive self-deception). Discuss “RSI progress” without specifying the level, and the conclusions are bound to be vague.\nThird, loop speed is a double-edged sword, and an amplifier of safety problems. Every paper on self-improvement talks about convergence speed, and few talk about runaway speed at the same time; yet the two share the same structure—the loop compounds “small selection biases”. For self-improving systems, what should be monitored is the scissors gap between in-loop and out-of-loop metrics, not any single metric.","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"My Three Takes","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"open-questions","content":"Whether there is a systematic way to construct task transformations for open-ended tasks (rather than hand-tuning them task by task); RLSVR only provides the paradigm; Where the capability ceiling of the “don’t change the weights” route lies—whether there are capabilities that can only be obtained by changing the weights; there is no clean evidence yet; An evaluation protocol for cross-loop transferability: who evaluates, with what data, and how to keep the evaluation from being contaminated by any one of the loops.","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"Open Questions","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"references","content":"Notes on a conversation with Ding Duo: 《在模型持续进步的时代，创业者应该站在哪里》§20 (the two routes for RSI and continual learning, and the three hard problems) RLSVR: arXiv:2607.23802 (abstract and core claims) WebSailor: arXiv:2507.02592, V2 2509.13305 (titles and positioning within the series); DeepResearcher arXiv:2504.03160 Recursive reward modeling: Leike et al. arXiv:1811.07871; Scalable Oversight arXiv:2211.03540 (hands-on-modern-rl textbook: docs/chapter30_alignment_failures/scaling-and-defenses.md) Classic Test-Time Training: Sun et al. arXiv:1909.13231; test-time scaling: s1 arXiv:2501.19393 Previous post: Who’s to Blame When a 7-Turn Interaction Fails: Reward Credit Assignment for Multi-Turn Trajectories (previous post: within-episode credit assignment; this post: cross-episode loops)","date":"2026-09-01","permalink":"https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/","section":"References","summary":"RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.","tags":["rl","post-training","agentic-rl","rsi","self-improvement"],"title":"“The Model Trains Itself”: RSI and Three Loops"},{"anchor":"","content":"In single-turn RL the whole sequence shares one advantage, and nobody has ever lost sleep over it; in multi-turn agentic RL, the same trajectory can get the turn-1 search right, send the turn-2 query off course, and end with a wrong answer—so how to split a single trajectory reward becomes a real design decision. This post unpacks the formalism (action mask, trajectory probability), the credit-assignment spectrum (ORM/PRM/turn discounting/three middle-ground routes), and the opening GSPO-token leaves for multi-turn training.\nThis post comes with a companion interactive report (in Chinese), featuring a toggleable ORM/PRM credit-assignment comparison, backward discounting with a γ slider, and a breakdown of the GSPO-token formula.","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"a-problem-that-doesnt-exist-in-single-turn-training","content":"Consider a five-turn trajectory: turn 1’s search hits the key evidence; turn 2 drifts off course while rewriting the query and brings back irrelevant content; turn 3 corrects course and gets halfway back; turn 4 makes a calculation error; turn 5 gives a wrong answer. The environment returns just one number: failure, reward = 0.\nSingle-turn RL doesn’t have this problem—the entire response shares one advantage, and “all right or all wrong” is taken for granted. In the multi-turn setting this crude logic breaks down immediately: turn 1 clearly did the right thing, so why should it be penalized along with turn 4? On top of that there is a further engineering layer: the trajectory mixes two kinds of tokens, those the model generated itself (thinking, tool-call arguments, the final answer) and those returned by the environment (search results, code output), and the latter should never enter the gradient at all. So “how does turn-level reward get into GRPO/GSPO” is really three questions rolled into one: which tokens count as actions, how much advantage each turn gets, and which tokens that advantage gets multiplied onto. This post takes them apart in that order.","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"A Problem That Doesn't Exist in Single-Turn Training","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"first-pin-down-the-object-trajectory-probability-and-the-action-mask","content":"In the formalization of multi-turn trajectories (the textbook adopts the POMDP formulation from the AppWorld paper, arXiv:2504.11536), the full state is z_t = [s_0, c, x_{1:t}]: s_0 is the hidden initial environment state (database snapshot, REPL state), c is the task context, and x_{1:t} is the entire token history so far—both what the model generated and what the environment returned. The model sees only the text part, which is why it is called partially observable.\nThe chain-rule factorization of the trajectory probability has one key detail:\n\\rho_\\theta(x \\mid s_0, c) = \\mathbb{I}(s_0, x) \\prod_{t \\in a(x)} p_\\theta(x_t \\mid c, x_{1:t-1})\nThe product runs only over a(x), the set of positions of model-generated tokens; the observation tokens returned by the environment are determined by the environment dynamics and are not part of the product. Translated into the loss, this is the action mask: a 0/1 vector the same length as the trajectory, with model tokens marked 1 and the prompt and tool returns marked 0. SearchR1’s implementation makes this very explicit: the contents of \u003cthink\u003e and \u003csearch\u003e take part in training, while the \u003cinformation\u003e segments returned by the retriever are masked out entirely. Letting environment tokens into the gradient as well would amount to forcing the model to “learn to predict which web pages the search engine returns”, contaminating the policy gradient.\nThis sets the skeleton of this post’s answer: turn-level reward enters training not by weighting environment tokens, but by constructing a separate advantage for each turn’s model tokens.","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"First, Pin Down the Object: Trajectory Probability and the Action Mask","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"three-layers-of-signal-how-reward-travels-from-trajectory-to-weights","content":"The textbook splits the signal flow into three layers, and this layering is the map for understanding the whole spectrum of approaches:\nTrajectory reward R(\\tau_i): the raw scalar given by the environment—“did this episode succeed or not”; Step advantage A_{i,t}: breaks the final outcome down into per-turn scores, answering “should turn t be rewarded”—this is the main battleground of credit assignment; Token gradient: A_{i,t} is multiplied onto the log-probs of that turn’s action tokens, \\Delta\\theta = A_{i,t} \\sum_{k \\in a_{i,t}} \\nabla \\log \\pi_\\theta(y_k \\mid \\cdot). What “putting turn-level reward into GRPO” actually comes down to is layer 2: GRPO’s within-group comparison naturally works at the trajectory level (ranking multiple trajectories for the same problem against each other), so to get down to the turn level you have to define A_{i,t} yourself and feed it in.","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"Three Layers of Signal: How Reward Travels from Trajectory to Weights","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"why-the-naive-approach-fails-a-controlled-experiment","content":"The textbook’s Mini Agent Loop experiment gives the most convincing set of numbers. The task: look up Earth’s radius → compute the equatorial circumference → verify → answer. Construct a “bad” trajectory: only step 2 is wrong (π taken as 3), steps 1 and 3 are both correct, and the final answer is wrong.\nORM (final outcome only): on failure, every step’s credit is 0—because 0 \\times \\gamma = 0, after the zero signal propagates backward every step gets “no signal”; if you switch to “-1 on failure”, the correct search in step 1 receives a penalty of -0.857. Either the innocent get blamed, or everyone simply goes silent. PRM (each step scored independently): correct steps get positive scores and wrong steps negative ones; the discriminability is 19 times that of ORM, and at the same number of steps the task success rate is about 30 percentage points higher. The cost is that every step has to be evaluated—which in real settings means annotation cost or training yet another scoring model. PRM is too expensive and ORM too coarse; the middle-ground routes of 2025–2026 fall into three categories by “where the signal comes from”, and all of them are essentially looking for a compromise between the two.","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"Why the Naive Approach Fails: A Controlled Experiment","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"the-credit-assignment-spectrum-three-routes-between-orm-and-prm","content":"Route 1: compare actions from the same state (state-anchored). GiGPO’s idea is that if multiple rollouts within a group converge on some state s, you compare “the subsequent returns of the different actions taken from the same situation”, and the within-group relative score is the step advantage. Its refined version, HGPO, points out that the same page is not the same context (depending on whether a constraint was missed earlier, the same “add to cart” can mean something completely different), so it stratifies the comparison groups by k-step history and fuses the strata with weights. This route needs no extra annotation at all, but it depends on states being able to collide—web tasks are naturally friendly to it, while in open-ended dialogue collisions almost never happen.\nRoute 2: redistribute the final reward back to each step (progress redistribution). SPA-RL trains a progress estimator under the constraint that the per-step contributions sum back to the final reward (\\hat R = \\sum_t \\hat c_t \\approx R): the contributions along a successful trajectory sum to nearly 1 and along a failed one to nearly 0, which forces the model to assign positive contribution to the steps that actually push the task forward. IGPO is cheaper: it directly uses “the increase in the model’s log probability of the ground-truth answer” as the turn-level reward—if after turn t the model is more confident in the correct answer (log-prob rises from -4.0 to -1.5), it records r^{IG} = +2.5; if the model was led astray, it records a negative value. What it yields is a ready-made turn-level scalar that can go straight into discounted accumulation, with no Monte Carlo value estimation needed; the cost is a dependence on high-quality reference answers, and questions with multiple valid answers get unfairly penalized.\nRoute 3: change the optimization granularity (the step-aligned paradigm). StepPO and Turn-PPO take the most radical view: existing algorithms inherit RLHF’s token-centric paradigm, but an agent’s decisions are naturally turn-grained, so there is a granularity mismatch between token-level optimization and turn-level decisions. Turn-PPO uses PPO instead of GRPO for multi-turn training, on the grounds that trajectory-level within-group advantages have too much variance, and switches to turn-level advantage estimation—each turn has its own advantage, shared among the tokens within that turn. This route is orthogonal to all the methods above: the others change where A_{i,t} comes from; this one changes the basic unit of optimization.\nWhichever route you take, multi-turn RL has one more problem, along a purely temporal dimension: the earlier the mistake, the greater the responsibility (if step 1 goes wrong, every later step unfolds on top of the error). Both the textbook and engineering implementations handle this with backward discounting: working back from the final outcome, the earlier the step, the larger the discount—for a 7-turn interaction in which only the last turn has reward 1.0 and \\gamma = 0.9, the returns assigned to the turns are [0.531, 0.590, 0.656, 0.729, 0.810, 0.900, 1.000].","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"The Credit-Assignment Spectrum: Three Routes Between ORM and PRM","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"where-the-grpogspo-family-sits-on-this-map","content":"This post settles the loose end left by the GSPO post. GSPO’s principle that “the unit of optimization should match the unit of reward” runs into real tension in multi-turn settings: GSPO’s objective treats the entire response as a single action, which suits outcome-level rewards; multi-turn RL wants turn-level credit assignment, which amounts to demanding a granularity finer than the sequence. The GSPO-token variant that §4.3 of the GSPO paper designs for exactly this has a very telling formula:\ns_{i,t}(\\theta) = \\mathrm{sg}[s_i(\\theta)] \\cdot \\frac{\\pi_\\theta(y_{i,t}|x,y_{i,\u003ct})}{\\mathrm{sg}[\\pi_\\theta(y_{i,t}|x,y_{i,\u003ct})]}\n\\mathrm{sg}[\\cdot] is stop-gradient (it takes only the value and passes no gradient back). The beauty of this construction: numerically, s_{i,t} is identically equal to the sequence-level ratio s_i, so the objective, the clipping condition, and the theoretical gradient are fully equivalent to GSPO’s; yet the gradient expression now contains a “per-token customizable advantage slot”. Set all tokens of the same turn to the same A_{i,t} and you get exactly turn-level credit assignment—this is a formal opening the algorithm layer leaves for multi-turn training, not an engineering hack.\nThe GRPO family takes the other, symmetric route: it leaves the mechanism of comparing multiple trajectories within a group untouched and outsources the construction of A_{i,t}—whether you hand it trajectory-level comparisons (the default) or turn-level scores (ORM + discounting / PRM / IGPO / SALT), it multiplies them in all the same. So in implementation terms, “turn-level reward in GRPO” is not a change to the algorithm but a change to the advantage constructor + making sure the action mask is clean. The combination of the two routes at the paper level (GSPO-token × turn-level A_{i,t}) currently has no public ablation—a genuine gap.","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"Where the GRPO/GSPO Family Sits on This Map","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"my-three-takes","content":"First, the number of turns sets the tier; the algorithm can’t. The textbook’s selection guide tiers by turn count: for 3–5 turns, plain ORM/GRPO is enough (episodes are short, so sparsity isn’t fatal); for 5–15 turns, bring in milestone rewards or annotation-free methods like SALT/GiGPO; beyond 15 turns, PRM or progress rewards plus tree search are a must. This tiering matters more than any single technique—when reward density can’t sustain learning, switching to a stronger RL algorithm yields zero gain. The five-turn trajectory from the top of this post actually sits right on the boundary between the first and second tiers.\nSecond, the link between credit-assignment quality and entropy is more direct than you might think. Under ORM every step of a failed trajectory gets penalized, and the easiest countermeasure the model learns is to reduce behavioral diversity (take fewer actions, answer earlier)—in the multi-turn setting, entropy collapse is not a by-product but a direct consequence of wrong credit assignment. Conversely, fine-grained signals like PRM or IGPO let the model “keep the right steps and fix only the wrong ones”, which is what preserves exploration. In multi-turn training, watching the agent’s turn-count distribution and tool-call diversity exposes credit-assignment problems earlier than watching the reward curve.\nThird, four reward types stand between “it runs” and “the credit is assigned correctly”. The textbook divides agentic rewards into four categories: Outcome (is the final result correct), Format (can the action be parsed and executed), Cost (number of turns, API spend), and Process (does the step advance the task). When getting started, people usually implement only the first two, to make sure training runs, and Process is exactly what the credit-assignment layer needs to make dense. But many engineering setups also fold the Format penalty into the outcome signal and assign credit for both together—a turn with a format error and a turn with a reasoning error bear the same blame in the trajectory, and almost no public report teases this kind of conflation apart.","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"My Three Takes","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"open-questions","content":"There is no public ablation of how GSPO-token combined with turn-level A_{i,t} performs on multi-turn agentic tasks—whether Qwen has validated it internally is impossible to know; The \\gamma of turn-level discounting has not been systematically swept in multi-turn LLM training (the textbook example uses 0.9, which comes from classical RL convention rather than an ablation); The mix and isolation of the four reward types (outcome/format/cost/process) at the credit-assignment layer lack public controlled experiments—most reports give only final metrics, not the credit-assignment structure.","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"Open Questions","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"references","content":"Textbook hands-on-modern-rl: 22.2 Multi-turn RL formalization (AppWorld POMDP, action mask, the four reward types) and 22.3 Trajectory credit assignment (three signal layers, the ORM/PRM experiment, three middle-ground routes, turn-level discounting, selection tiers), in docs/chapter22_agentic/formulation.md and docs/chapter22_agentic/credit-assignment.md GSPO paper: arXiv:2507.18071 §4.3 (the GSPO-token formula and the equivalence argument) AppWorld: AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents; original ORM/PRM framework: Let’s Verify Step by Step (Lightman et al., 2023) Middle-ground routes: SALT [2510.20022], GiGPO [2505.10978], HGPO [2602.22817], SPA-RL [2505.20732], IGPO [2510.14967], Turn-PPO [2512.17008], StepPO (the ID given in the textbook points to a different paper; the original could not be verified) (arXiv IDs checked against arXiv) Previous posts: The Ratio’s Wrong Unit: GSPO Goes Sequence-Level, Entropy Collapse: What DAPO’s Clip-Higher Fixes","date":"2026-08-31","permalink":"https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/","section":"References","summary":"Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.","tags":["rl","post-training","agentic-rl","grpo","credit-assignment"],"title":"A 7-Turn Failure: Assigning Blame in GRPO/GSPO"},{"anchor":"","content":"In the same batch, which contributes more to the gradient: a 500-token response or a 5000-token one? GRPO says they count equally, DAPO says the long one counts ten times as much, and both can justify their answer. This is a dispute over loss aggregation: sample-mean gives each response one vote, token-mean gives each token one vote. The DAPO paper (§3.3) says long-CoT settings must switch to token level, but the switch pushes the length skew from the response level to the question level. This post unpacks the three formulas, the two camps’ arguments, and a tradeoff few people talk about.\nThis post has a companion interactive report (in Chinese), with a weight calculator that switches among the three vote-counting rules and a side-by-side breakdown of the formulas.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"One Vote or Ten: token-mean vs sample-mean","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"first-count-the-votes","content":"The same training batch contains two responses: one is 500 tokens, the other 5000 tokens. How much does each contribute to this step’s gradient?\nGRPO (sample-mean): equally. First average over the tokens within each response, then average over the G responses — every response, long or short, gets one equal vote; DAPO (token-mean): the long one contributes ten times as much. All tokens in the batch are flattened together and divided by the total token count — one vote per token, so the 5000-token response gets 5000 votes; Dr.GRPO: formally also one vote per response, but the divisor changes from “each response’s actual length” to a global constant, which changes what a vote means (that is the topic of the Dr.GRPO post). All three answers are backed by papers, and all three camps are working on the same spot in the same expression — the loss’s aggregation divisor. This knob is orthogonal to the advantage normalization (the std half) covered in the Dr.GRPO post: that post’s question was “how is the advantage computed?”; this post’s question is “once it is computed, by what rule are the tokens’ losses added together?”","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"First, Count the Votes","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"three-formulas-side-by-side","content":"The GRPO objective (DeepSeekMath’s Equation 3, with the KL term omitted):\n\\mathcal{J}_{GRPO} = \\mathbb{E}\\left[\\frac{1}{G}\\sum_{i=1}^{G}\\frac{1}{|o_i|}\\sum_{t=1}^{|o_i|}\\ell_{i,t}\\right], \\quad \\ell_{i,t}=\\min\\big(w_{i,t}\\hat{A}_i,\\ \\mathrm{clip}(w_{i,t},1-\\varepsilon,1+\\varepsilon)\\hat{A}_i\\big)\nHere o_i is the i-th response, |o_i| is its length, w_{i,t} is the probability ratio between the new and old policies for that token, and \\hat{A}_i is the advantage shared by the entire response. Note the structure of the sums: \\frac{1}{|o_i|}\\sum_t first averages each response on its own into a single number, and \\frac{1}{G}\\sum_i then averages those G numbers. The two responses each cast one vote, regardless of length.\nDAPO changes just one spot (Equation 12 of the paper, the part highlighted in red):\n\\mathcal{J}_{DAPO} = \\mathbb{E}\\left[\\frac{1}{\\sum_{i=1}^{G}|o_i|}\\sum_{i=1}^{G}\\sum_{t=1}^{|o_i|}\\ell_{i,t}\\right]\nThe inner \\frac{1}{|o_i|} is gone, the two levels of summation are flattened into one, and the divisor becomes the total token count of the whole group. Every token gets exactly the same weight, 1/\\sum_i|o_i|; a response’s total weight is proportional to its length.\nThe third is Dr.GRPO: the same aggregation structure as DAPO (tokens flattened), but the divisor is a global constant (such as the length cap), and the std is removed from the advantage as well — the Dr.GRPO post went into that in detail; this one looks only at the divisor.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"Three Formulas Side by Side","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"why-dapo-insists-on-switching-two-places-the-penalty-cant-reach","content":"The argument in §3.3 of the DAPO paper is quite concrete and targets the long-CoT setting. Under sample-mean, long samples “contribute disproportionately little to the overall loss” (original wording: disproportionately lower contribution), which leads to two bad outcomes:\nFirst, reasoning patterns in high-quality long samples don’t get learned. In a correct 5000-token chain of reasoning, each token receives a gradient weight of 1/(G\\cdot 5000), only one-tenth of what a token in a 500-token short sample gets. What long samples carry is precisely complex reasoning, so under sample-mean, the more important the content, the lower its per-unit weight.\nSecond, bad patterns in overlong samples don’t get penalized away. The paper reports an observation: overly long samples often contain low-quality patterns such as gibberish (garbled text) and repeated words. The penalty on these bad tokens is likewise diluted by 1/|o_i| — the longer the sample, the more lightly each bad token is penalized. The result is “an unhealthy increase in entropy and response length” (the measured curves in Figures 4a/4b of the paper).\nAfter switching to token-mean, a token carries the same weight whether it sits in a long sample or a short one. DAPO is upfront about the effect: as a result, longer sequences “have more influence on the overall gradient update than shorter ones” (original wording: longer sequences can have more influence). This is not a side effect but a design goal — in the long-CoT era, the gradient should be spent mainly on long responses in the first place.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"Why DAPO Insists on Switching: Two Places the Penalty Can't Reach","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"a-tradeoff-few-people-talk-about","content":"But the phrase “long samples get more votes” deserves a pause and a second look: it has two layers of meaning, and DAPO only spells out the first.\nThe first layer is the response level: within a single response, tokens are no longer diluted by length. This is a direction Dr.GRPO endorses too — the “response-level length bias” that Dr.GRPO criticizes (when correct, shorter responses are favored; when wrong, longer ones are safer) and the “bad tokens don’t get penalized” problem that DAPO criticizes are two faces of the same 1/|o_i|, and the two camps’ prescriptions reach the same destination by different roads: equal weight per token.\nThe second layer is the question level: across the whole batch, questions with long responses take more of the total weight. A question on which the model’s responses average 6000 tokens and one on which they average 600 tokens differ in batch weight by 10x under token-mean; under sample-mean the two are equal. In other words, token-mean pushes the length skew from the response level to the question level — long questions dominate the gradient. In the long-CoT setting this may be exactly what you want (hard, reasoning-heavy questions tend to have long responses too), but it is a real bias: question weights are no longer uniform; instead, each question is overweighted in proportion to its current length. I have not seen this layer of the tradeoff quantified in any paper (this point is an inference; happy to be proven wrong).\nThere is also a second second-order effect: DAPO’s divisor \\sum_i|o_i| is a random variable that varies from batch to batch, whereas Dr.GRPO’s constant divisor is not. The direct consequence of a random divisor is that the gradient scale fluctuates with the length composition of each batch; the indirect consequence is more interesting — later in training, responses generally get longer, the divisor grows, and each token’s effective learning rate decays automatically. In other words, token-mean comes with a built-in implicit learning-rate decay that moves in lockstep with length inflation, a property that neither sample-mean nor a constant divisor has. This is a mechanism-level inference that no paper has measured, but it is a real variable when choosing an aggregation method: what you are choosing is not just a weight allocation but also the dynamics of the gradient scale.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"A Tradeoff Few People Talk About","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"both-say-back-to-ppo-but-not-the-same-ppo","content":"An easily overlooked historical fact: PPO’s original objective was aggregated per token to begin with (summing or averaging over the tokens of the whole batch, with GAE supplying a per-token advantage). It is GRPO’s sample-mean that is the outlier — because GRPO’s advantage is “one number per response,” averaging each response into one number first and then averaging across responses is semantically the tidiest.\nThen DAPO and Dr.GRPO both claim to have “restored the PPO objective”: DAPO restores PPO’s aggregation scheme (tokens flattened, but the advantage keeps its z-score, so the std is still there); Dr.GRPO restores the form of PPO’s objective function (a constant divisor, the advantage reduced to r-\\text{mean}, the std removed). The two knobs — aggregation granularity and advantage normalization — are independent and can be combined freely: in implementations such as verl, the two really are configured separately (from the implementation-level discussion in the Dr.GRPO post: by default, masked_mean in trl/verl adds back the per-response averaging that the formula doesn’t have, and auditing whether each divisor “is a variable or a constant” is the cheapest check there is). This also means that “GRPO vs DAPO vs Dr.GRPO” is not three algorithms lined up side by side, but a configuration space of at least 2×2, and the good points known so far are all still being found by trial and error along its edges.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"Both Say \"Back to PPO,\" but Not the Same PPO","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"which-vote-should-weigh-more-and-when","content":"Break it down by scenario and the choice isn’t mysterious:\nShort CoT, low length variance: the two aggregations are approximately equivalent (lengths are similar, so 1/|o_i| and 1/\\sum|o_i| differ by a roughly constant factor), so there’s no need to agonize; Long CoT, bad patterns hiding in long samples (repetition, gibberish, pointless rambling): token-mean is crucial, otherwise bad tokens will never be penalized enough — this is DAPO’s core scenario; You want uniform weights across questions (e.g., question difficulty varies widely and you worry about long questions dominating): sample-mean or a constant divisor is more controllable; You want a stable gradient scale and predictable learning-rate behavior: a constant divisor (the Dr.GRPO route) is the cleanest; token-mean’s random divisor mixes length dynamics into the learning rate. My take: at its core, the aggregation dispute is about “in what unit the gradient budget is allocated.” Under sample-mean it is allocated per response, implicitly assuming every response carries the same amount of information — true in the era of short answers; under token-mean it is allocated per token, implicitly assuming every token carries the same amount of information — closer to the truth in the long-CoT era, but at the cost of letting length decide the weighting across questions. The paper that is really missing is one that makes this budget allocation explicit: weights assigned neither per response nor per token, but according to estimated information content or uncertainty. Until then, DAPO’s change is the most pragmatic default for long CoT.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"Which Vote Should Weigh More, and When","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"open-questions","content":"The effect of token-mean’s question-level length skew (cross-question weight ∝ length) on where training ends up has not been publicly quantified; Nobody has measured how the implicit learning-rate decay from the random divisor interacts with an explicit LR schedule stacked on top of it; For a full 2×2 ablation of the two knobs, aggregation granularity and advantage normalization (sample/token × with std/without std), the public literature only has the points on the diagonal.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"Open Questions","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"references","content":"DAPO paper: arXiv:2503.14476; token-level loss in §3.3 (Equation 12), entropy and length curves in Figure 4 Dr.GRPO paper: arXiv:2503.20783; the original statements of the response-level length bias and the question-level difficulty bias DeepSeekMath / GRPO: arXiv:2402.03300; the sample-mean objective (Equation 3) Previous post (covering the advantage-normalization half): Dr.GRPO: Two Normalization Terms, Two Biases (it gave the token-mean section number as §3.2 by mistake; the correct one is §3.3, and that has been corrected in passing) The question-level length skew and the random divisor’s implicit LR decay are this post’s own inferences, each flagged where it appears in the text","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/token-mean-vs-sample-mean/","section":"References","summary":"Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.","tags":["rl","post-training","grpo","dapo","dr-grpo"],"title":"One Vote or Ten: token-mean vs sample-mean"},{"anchor":"","content":"RL training needs a generation length cap, and truncated responses are treated as wrong by default—which misjudges ‘undecidable’ samples as ‘wrong’ ones and injects reward noise straight into the group-relative advantage. DAPO’s two fixes: Overlong Filtering acknowledges that such samples are undecidable and masks their loss; Soft Overlong Punishment builds a linear buffer ramp before the cliff, turning length control into a continuous signal. This post unpacks the mechanism, the formula, and the configuration values, and fits it together with Dr.GRPO and token-mean into a complete picture of the length problem.\nThis post comes with a companion interactive report (in Chinese), featuring a drill-down length ruler, a three-piece breakdown of Equation 13, and a switchable comparison of the three handling approaches.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"Judging Truncated Answers: DAPO's Overlong Fix","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"an-engineering-setting-you-cant-avoid","content":"Rollouts in RL training must have a generation length cap. GPU memory has to be reserved for the longest sequence, the batch has to wait for the slowest response to finish, and a single runaway response can drag down the throughput of the whole batch. DAPO’s configuration: an expected maximum length of 16,384 tokens, an additional 4,096-token buffer, and a hard generation cap of 20,480 (paper §4.1).\nOnce a cap is set, some responses will inevitably be truncated. What should happen to these samples? The default practice is common: assign a punitive reward and treat them as wrong answers (reward recorded as 0 or a negative score).\nThe DAPO paper (arXiv:2503.14476) points out in §3.4 that this default causes problems: improper handling of truncated samples introduces reward noise, and that noise is injected into the entire training process, amplified by within-group normalization. The paper’s reasoning is straightforward: when a reasoning process that is sound in itself gets penalized merely for running too long, the penalty confuses the model—it can’t tell whether its reasoning was valid.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"An Engineering Setting You Can't Avoid","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"how-the-noise-gets-into-the-advantage","content":"GRPO’s advantage is a within-group relative ranking: sample G responses for the same problem and subtract the group mean (dividing by the std). This means that once the reward of any single response is wrong, it affects the whole group.\nTake a concrete example: 8 responses are sampled for a problem, and the true split is 4 right and 4 wrong; among them, 2 “wrong” responses are actually high-quality reasoning that got truncated (had they been finished, they would likely have been right).\nBy true quality: the group mean is 0.5, the correct responses get positive advantages, and the wrong ones get negative ones; Scoring truncation as 0: the truncated samples are also counted as wrong answers and the group mean is dragged down (6 counted as wrong, mean 0.25), so the advantage of every correct sample is inflated and the negative advantage of every wrong sample is softened—including the two that genuinely got it wrong. As if one layer of distortion weren’t enough, there is a second: those two truncated high-quality reasoning traces themselves get reinforced as negative samples. What the model learns from them is not “write a bit shorter” but “my way of reasoning is wrong”—which is exactly where the paper says the model gets confused. Once these two kinds of penalty are lumped together, the model throws out the good way of reasoning along with “writing long”.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"How the Noise Gets into the Advantage","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"approach-1-overlong-filteringdeem-truncated-samples-undecidable-and-mask-their-loss","content":"DAPO’s first approach: rather than treating a truncated sample as a wrong sample, simply refuse to recognize it as a sample at all. Overlong Filtering masks the entire loss of a truncated response—the sample takes no part in advantage estimation and produces no gradient.\nThe effect shown in Figure 5 of the paper: after applying Overlong Reward Shaping, both AIME accuracy and the actor model’s entropy become more stable, and training stability improves markedly. This is worth noting: the stabilized entropy means truncation noise itself is interfering with the exploration dynamics—it doesn’t merely affect the reward numbers on paper.\nThe costs are just as obvious. The paper doesn’t go into them, so I’ll add two points:\nWasted compute. The rollout cost of a truncated sample has already been paid (it is often the longest response, which happens to consume the most compute), and once it is masked, not a single gradient comes back in return. This is the same logic as dynamic sampling—both “trade compute for signal quality”, which is the orientation of the whole DAPO paper; Hidden survivorship bias. Which samples are more likely to be truncated? Long problems, hard problems, problems that need long CoT. Systematically masking truncated samples amounts to systematically down-weighting hard problems in the training distribution. In the short term, training becomes more stable; the long-term cost is that the model may reveal weaknesses on exactly the problems that most need long reasoning. The paper doesn’t discuss this side effect; this reservation is my own.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"Approach 1: Overlong Filtering—Deem Truncated Samples \"Undecidable\" and Mask Their Loss","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"approach-2-soft-overlong-punishmentbuild-a-buffer-ramp-before-the-cliff","content":"Masking everything loses information: the model can’t learn that length itself matters. DAPO’s second approach is Soft Overlong Punishment, which turns length itself into a continuous, discriminable reward dimension (Equation 13 in the paper):\nR_{length}(y) = \\begin{cases} 0, \u0026 |y| \\le L_{max} - L_{cache} \\\\ \\dfrac{(L_{max} - L_{cache}) - |y|}{L_{cache}}, \u0026 L_{max}-L_{cache} \u003c |y| \\le L_{max} \\\\ -1, \u0026 L_{max} \u003c |y| \\end{cases}\nThe three pieces, one at a time:\n|y| \\le L_{max}-L_{cache} (i.e., ≤ 12,288 tokens under DAPO’s configuration): no penalty. Length is completely free in this regime; Entering the cache interval (12,288 to 16,384): a linearly ramping penalty, falling smoothly from 0 to -1. Every extra token written increases the penalty by 1/L_{cache} = 1/4096; Beyond L_{max} (truncated): the penalty is capped at -1. Note that this term is added to the original correctness reward—a response that is correct but runs over can still have its total score pulled down. The most ingenious thing about this design: the cliff is not removed; a buffer ramp is built in front of it. Going past L_{max} is still a hard -1 penalty, just as harsh as the naive approach; but the model starts receiving a continuous “almost over budget” signal four thousand tokens before the cliff, and the signal strength is monotonically related to how far over budget it is. After within-group normalization, two responses—“12,500 tokens, nearly over” and “16,300 tokens, about to go over”—get a distinguishable advantage gap, from which the model can learn to wrap up early instead of only feeling the pain once it hits the cap. Length control turns from a binary event (not over / over) into a continuous, learnable task.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"Approach 2: Soft Overlong Punishment—Build a Buffer Ramp Before the Cliff","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"placing-it-in-the-full-length-problem-picture","content":"The previous few posts in this series all had to do with length; this post covers the final aspect:\nGradient dilution from 1/|y| (Dr.GRPO, see the normalization-bias post): the longer a wrong response, the smaller the penalty on each token, so the model tends to make wrong responses overly long. Fix: change the normalization term; token-mean (DAPO §3.3): instead of averaging within each sample first and then summing over the group, average uniformly over all tokens in the batch, which keeps long responses from taking an outsized weight in the loss. Fix: change the loss aggregation; Overlong (DAPO §3.4, this post): the truncated samples produced by the generation length cap have undecidable rewards. Fix: mask them (filtering) or change the reward function (soft punishment). The three address length problems at three different levels: gradient weighting, loss aggregation, and reward judgment. DAPO uses both token-mean and overlong shaping, meaning it fixes both the gradient side and the reward side; Dr.GRPO focuses on normalization. They don’t conflict.\nOne can also see a design orientation in DAPO: better to use no signal than an unreliable one. Dynamic sampling filters out prompts with no gradient signal (all-correct / all-wrong problems), overlong filtering masks samples with unreliable rewards (truncated responses), and Dataset Transformation converts all answers to integers to eliminate parser noise (§3.5, DAPO-Math-17K). All four techniques point in the same direction: training collapse is often not because the algorithm isn’t good enough, but because noise has crept into the signal.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"Placing It in the Full \"Length Problem\" Picture","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"my-three-takes","content":"First, truncated samples are “undecidable” samples, and judging them as wrong is a misjudgment at the reward level. The premise of RLVR is that rewards are verifiable—you assign a score only if you can judge right or wrong. For a truncated sample, the verifier never got to finish its judgment at all; it is neither a 1 nor a 0. Overlong Filtering acknowledges this undecidability, while Soft Punishment introduces a new decidable dimension (length itself). Both approaches are consistent with the principle that “rewards must be verifiable”, whereas the naive punitive reward violates it.\nSecond, the fact that entropy is more stable after overlong handling matters more than it appears. Figure 5(b) shows that the actor’s entropy curve is smoother after shaping. This indicates that the chaos near the length cap (high-quality long reasoning that suddenly gets penalized, responses forced into hasty endings) doesn’t merely distort the reward; it also adds exploration noise. Read this together with the Clip-Higher post (entropy collapse): entropy can collapse when wrongly suppressed by the optimizer, and it can also swing wildly when disturbed by reward noise. Entropy monitoring has to be analyzed together with the quality of reward judgments.\nThird, the cache width and the masking ratio are two hyperparameters that need monitoring, and there is currently no public basis for them. The paper doesn’t say how the buffer-ramp length of 4,096/16,384 = 25% was chosen; during training, the truncation rate (the fraction of masked samples) drifts with the model’s length behavior—when the masking rate is too high, the effective batch shrinks and the share of hard problems drops; when it is too low, that indicates length control has already broken down or the cap is set too loose. The DAPO paper has no monitoring curves for either number, so reproductions have to fill them in themselves.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"My Three Takes","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"open-questions","content":"There are no public experiments on whether the hard-problem weight lost to Overlong Filtering needs to be compensated with a progressive length curriculum in long training runs; The rationale and ablations for the choice of cache width (25%) are not disclosed in the paper; There is currently no data on whether long but correct exploratory responses get doubly suppressed when Soft Punishment’s -1 cap is stacked with Clip-Higher’s asymmetric clipping.","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"Open Questions","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"references","content":"DAPO paper: arXiv:2503.14476v2 (source of the §3.4 mechanism description, Equation 13, the §4.1 configuration values, and Figure 5) Textbook hands-on-modern-rl: Chapter 18 docs/chapter18_grpo/deepseek-dapo.md and the appendix DAPO cheatsheet (docs/en/appendix_code_cheatsheet/dapo.md) Previous posts: Entropy Collapse: What DAPO’s Clip-Higher Fixes, “When Wrong, Write Longer”: How GRPO’s Two Normalization Terms Distort the Optimization Direction","date":"2026-08-30","permalink":"https://njx-njx.github.io/en/posts/dapo-overlong-reward-shaping/","section":"References","summary":"Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Judging Truncated Answers: DAPO's Overlong Fix"},{"anchor":"","content":"R1-Zero’s “length and reward rising together” has been treated as the signature curve of emergent reasoning, but Sea AI Lab’s Dr.GRPO paper points out that half of that rise may be the optimizer’s own preference: 1/|y| makes the model lean toward writing longer when it answers wrong, and dividing by std lets questions at the difficulty extremes take more of the weight. A mechanistic breakdown of two normalization terms and two biases, plus an implementation-level bias hidden in masked_mean.\nThis post has a companion interactive report (in Chinese), with drill-down worked examples of the biases, a side-by-side comparison of masked_mean implementations, and a decision matrix for choosing a fix.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"“Wrong? Write Longer”: What GRPO Normalizers Skew","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"an-over-trusted-curve","content":"The most famous plot from R1-Zero training is the one where response length and training reward rise in lockstep. The community was quick to spin a story around it: the model has learned to think longer, and long CoT keeps taking shape through RL.\nSea AI Lab’s Dr.GRPO paper (arXiv:2503.20783) poses a simpler question: on the plateau where reward has stopped rising, why does length keep growing? After pinning down the causes one by one, the paper concludes that this is not the model learning to reason more; rather, the optimizer keeps pushing the model to generate longer responses. The driving force comes from two normalization terms in the GRPO objective: they adjust the model’s optimization direction in opposite ways for correct and incorrect responses, and their effect on incorrect responses points to one and the same outcome: when the answer is wrong, the longer the response, the less it gets penalized.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"An Over-Trusted Curve","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"bias-1-1y--per-token-averaging-dilutes-the-penalty","content":"The GRPO objective (Equation 3 in the paper) contains an inconspicuous denominator:\n\\mathcal{J}_{GRPO} = \\mathbb{E}\\left[\\frac{1}{G}\\sum_{i=1}^{G} \\frac{1}{|y_i|}\\sum_{t=1}^{|y_i|}\\min\\left(w_{i,t}\\hat{A}_i,\\ \\mathrm{clip}(w_{i,t},1-\\varepsilon,1+\\varepsilon)\\hat{A}_i\\right)\\right]\nWhat 1/|y_i| means is this: within a single response, first average the gradient terms over all of its tokens, then add up the G responses. In other words, the gradient weight each token actually receives is \\hat{A}_i/|y_i|. A concrete worked example: sample two responses to the same question, one 100 tokens long and one 1000 tokens long, both with an advantage of absolute value 1 (opposite in sign):\nCorrect (\\hat{A}=+1): each token of the short response is reinforced by 1/100 = 0.01, while each token of the long response is reinforced by 1/1000 = 0.001. Among correct responses, the shorter, the more favored; Incorrect (\\hat{A}=-1): each token of the short response is penalized by 0.01, while each token of the long response is penalized by only 0.001. Among incorrect responses, the longer, the less penalized — a tenfold difference. This is not an anomaly caused by a math error; it is something the objective writes into every update step: correct answers must be concise; wrong answers may as well be long-winded. Late in training, once reward plateaus, the model’s room for improvement on the first half is used up, while the second half keeps pushing the model to generate longer responses — incorrect responses in particular keep getting longer. Fig.5 Plot 2 in the paper’s Figure 4 clearly shows this phase: GRPO’s length curve keeps rising after reward plateaus.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"Bias 1: 1/|y| — Per-Token Averaging Dilutes the Penalty","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"bias-2-dividing-by-std--extreme-difficulty-questions-take-more-weight","content":"The second normalization term sits in the advantage:\n\\hat{A}_i = \\frac{R_i - \\mathrm{mean}(\\{R_j\\}_{j=1}^G)}{\\mathrm{std}(\\{R_j\\}_{j=1}^G)}\nDividing by the within-group reward standard deviation is meant to put advantages on a common scale (a standard RL trick). But the paper points out a key difference: classic RL advantage normalization computes the std over the whole batch, with all samples sharing one divisor; GRPO computes the std separately for each question, so every question has its own divisor — and different divisors mean different weights.\nOne more worked example: for question 1, all 8 sampled responses are correct (std≈0.35, \\hat{A}≈+2.9 on the z-score scale); for question 2, the 8 samples are four correct and four wrong (std≈0.52, \\hat{A}≈±1.9). The more extreme the question — the all-correct giveaway, the all-wrong dead end — the smaller its std, and the larger its weight after the division. The model spends the most update budget on questions it already gets entirely right (the easy ones) and on questions it cannot solve at all (never answered correctly even once), while questions that are “partly right, partly wrong, right in the zone where learning pays off” get the least weight. This is what the paper calls difficulty bias.\nA caveat on scope: std normalization is not all downside. Its design intent, amplifying the signal in low-variance cases, is to let the model learn something even from questions it gets entirely right. What the paper criticizes is the unequal weighting caused by per-question normalization, not normalization itself.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"Bias 2: Dividing by std — Extreme-Difficulty Questions Take More Weight","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"bias-3-hidden-in-the-implementation-not-the-formula","content":"The third finding may hit engineers harder. The PPO objective (Equation 2 in the paper) sums over the response’s tokens, with no step that divides by length. But the paper examined mainstream open-source implementations and found that the losses in frameworks such as trl and verl all contain a masked_mean line: it averages using the response’s actual token count as the divisor (loss.sum(dim) / mask.sum(dim)). In other words, the formula has no length bias, but the implementation adds one.\nHow did this happen? The paper’s conjecture traces it to historical practice: in pretraining, all data is packed into fixed-length contexts, and loss.mean(-1), averaging over the context length, is both numerically convenient and stable; it is division by a constant and introduces no bias. Once this idiom was carried over to the RL stage, the divisor quietly changed from “the fixed context length” to “the actual length of each response” — from a constant to a variable, and the effect changed with it. Dr.GRPO’s implementation fix changes just one line: replace the divisor in masked_mean with a global constant (such as the generation length cap MAX_TOKENS).\nThe formula’s authors and the framework’s authors may be running the same set of tests, yet they end up with two different gradients. This kind of “formula–implementation mismatch” is more insidious than paper-level biases: you could read the paper ten times and never catch it; only reading the code reveals it.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"Bias 3: Hidden in the Implementation, Not the Formula","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"drgrpos-fix-and-its-justification","content":"The fix is simple: remove both normalizations, 1/|y_i| and std, so that the advantage only subtracts the group mean:\n\\tilde{A}_i = R_i - \\mathrm{mean}(\\{R_j\\}_{j=1}^G)\nAppendix A of the paper gives three theoretical justifications: with both terms removed, the objective reverts to the PPO objective (Equation 2), and the advantage is just the baseline-free Monte Carlo return plus an unbiased baseline; subtracting the mean introduces no bias — the baseline’s expected contribution to the gradient is identically zero, a standard result in policy gradients; and the corrected advantage differs from RLOO (Leave-One-Out) only by a constant factor G/(G-1), which can be folded into the learning rate and does not affect training dynamics.\nThe controlled experiments are run in the Oat framework on Qwen2.5-1.5B. The results fall into three layers:\nTraining dynamics: early on, both GRPO and Dr.GRPO show the R1-Zero-style joint rise of reward and length, but after reward plateaus GRPO’s length keeps climbing, while Dr.GRPO’s length stabilizes (Fig.5 Plot 1\u00262); Behavior: on benchmarks, the length of Dr.GRPO’s incorrect responses drops significantly — the change also mitigates overthinking (Fig.5 Plot 4); Reproducibility: after math pretraining, Llama-3.2-3B likewise shows GRPO’s length rising together with reward, and Dr.GRPO likewise curbs the continued growth in length (Fig.7). The same recipe also lets the 7B Qwen2.5-Math-7B, using MATH levels 3–5 and training for 27 hours on 8 A100s, reach 43.3% on AIME 2024 — the 7B SOTA at the time, and the whole recipe is extremely simple.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"Dr.GRPO's Fix and Its Justification","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"a-textbook-erratum-and-the-division-of-labor-with-the-previous-posts","content":"Section 16.4 of the textbook hands-on-modern-rl also covers this pair of biases, but its wording deserves a correction: it labels “the advantage gets amplified as std approaches zero” as “length bias”, and attributes length inflation to “dividing by std encourages the model to increase within-group variance”. Checked against the original paper, this account is off on two counts: std causes difficulty bias (unequal weights across questions), not length bias; and the direct source of length bias is 1/|y_i|, which has nothing to do with std and acts on the weight of each token within a response. The two normalizations and the two biases each exist independently — the value of the erratum is that choosing a fix depends on this distinction: removing std does not solve the length problem.\nThe recent posts in this series happen to tackle this at three different levels:\nDAPO’s token-mean (§3.3): it changes “average within each sample first, then average over the G samples” to “average all tokens in the batch uniformly by the total token count”, i.e. \\frac{1}{\\sum_i|y_i|}\\sum_i\\sum_t. All tokens are thus weighted by the total token count, which formally eliminates length bias — but DAPO’s advantage keeps the z-score (including std), so it only solves half the problem; GSPO: it uses a single ratio at the sequence level and no longer sums over tokens term by term in the objective, so there is no “divide each token by |y|” averaging step at all — it sidesteps this bias structurally; Dr.GRPO (this post): it leaves the structure of the objective unchanged and only removes the two variable divisors from the normalization terms.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"A Textbook Erratum, and the Division of Labor with the Previous Posts","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"my-three-takeaways","content":"First, an RL training curve must be screened for optimizer-induced false signals before anything else. When “length keeps rising” is cited by academia as headline evidence that “RL is working” and, at the same time, by engineering teams as an anomaly metric that “calls for intervention”, tracing it back to the root matters all the more. The paper also observes that for Qwen2.5-Math models used with a template, the model’s pre-RL ability is first damaged by the template and then rebuilt by RL (§3.3) — another class of factor that needs ruling out: part of the “RL gain” is really just ability that was there all along. Before analyzing an RL curve, rule out, one by one, the biases the optimizer introduces and the artifacts the evaluation setup creates.\nSecond, Dr.GRPO and dynamic sampling solve different problems; neither replaces the other. Removing std normalization corrects weight distortion (samples that should not be amplified receive too much weight); DAPO’s dynamic sampling (questions with std=0 are simply dropped and resampled) deals with the absence of gradient signal (when the responses are all correct/all wrong/all zero, the advantage is still 0 after subtracting the mean, and nothing is learned). Note that after removing std, a zero-signal question is still zero-signal — Dr.GRPO brings no gradient to such questions. One ensures correct relative weights within the samples, the other ensures the samples as a whole carry a learnable signal; a real training system needs both configured at once.\nThird, when checking a normalization, first check “whether the divisor is a variable or a constant”. The three biases here share one pattern: a normalization that looks like routine practice turns out, on closer inspection, to use a denominator that is not a constant — for 1/|y_i| the divisor is the length of each response, for std it is each question’s within-group spread, and for masked_mean it is the mask sum of the current batch. Going through every mean/divide in the training code and asking “can this divisor change from sample to sample?” is the cheapest taste audit there is.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"My Three Takeaways","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"open-questions","content":"How the reweighting of “extreme-difficulty questions” after Dr.GRPO removes std plays out in very long training runs (tens of thousands of steps): the time window of the paper’s 1.5B/7B experiments is relatively short; The effective-learning-rate conversion when token-mean (DAPO) and removing the variable divisors (Dr.GRPO) are stacked in the same training stack: there is no public, unified convention; A systematic check of “how much of the loss.mean(-1) habit from pretraining-era packing still lingers in RL frameworks”: so far there is only the paper’s spot check of trl/verl.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"Open Questions","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"references","content":"Sea AI Lab / NUS: Understanding R1-Zero-Like Training: A Critical Perspective (the source of this post’s formulas, experiments, and bias definitions; v2) Textbook hands-on-modern-rl: Section 16.4 (using the corrected wording) and the appendix DAPO cheatsheet (docs/en/appendix_code_cheatsheet/dapo.md, a side-by-side of DAPO’s z-score advantage and the token-mean loss granularity) RLOO reference: Back to Basics: Revisiting REINFORCE Style Optimization Previous posts: Entropy Collapse: What DAPO’s Clip-Higher Fixes, The Ratio’s Wrong Unit: GSPO Goes Sequence-Level, One Model, Two Probabilities: Cost of Mismatch","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/dr-grpo-length-and-difficulty-bias/","section":"References","summary":"Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.","tags":["rl","post-training","grpo","dr-grpo","normalization"],"title":"“Wrong? Write Longer”: What GRPO Normalizers Skew"},{"anchor":"","content":"DeepSeekMath samples 64 responses per question, a teaching implementation defaults to 8, and the latest paper claims even 2 is enough — a 32x spread on the same knob. G is GRPO’s only source of a baseline and its biggest compute lever, so what does it actually buy? This post unpacks G’s three roles: the sample size of the control variate, the coverage of groups with effective gradients, and the quality of the std-normalization denominator, plus why dynamic sampling can replace a larger G with filtering.\nThis post has a companion interactive report (in Chinese), with a zero-gradient-group probability calculator, a dual-curve baseline chart, and a side-by-side comparison of the three camps’ choices.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"Samples per Prompt: What GRPO's Group Size Buys","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"a-knob-with-a-32x-spread","content":"On every training step, GRPO samples a group of G responses for each prompt. How large this G should be is a question on which different camps give wildly different answers:\nDeepSeekMath (the original GRPO paper, arXiv:2402.03300 §4.2): 64. With a batch size of 1024, that means a single step covers only 16 prompts; The RLHF Book’s companion teaching code (code/policy_gradients/configs/): 8, the default across the whole lineup, from GRPO to GSPO and CISPO; The October 2025 paper “It Takes Two: Your GRPO Is Secretly DPO” (arXiv:2510.00977): 2 is enough, and the paper argues specifically that the “large groups are necessary” view is a misreading of why GRPO succeeds. A 32x gap. Worse, G is two things at once: it is GRPO’s only source of a baseline (there is no critic; the group mean is the baseline), and it is the biggest multiplier on training cost (rollouts are the main compute expense in RL training, and doubling G doubles that cost). One knob governs both signal quality and the compute bill, yet people have gotten everything from 2 to 64 to work — which suggests that the question “what does G actually buy?” is itself worth taking apart.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"A Knob With a 32x Spread","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"role-one-the-baselines-sample-size","content":"Having dropped the critic, GRPO uses the group mean as its baseline: \\hat{A}_i = r_i - \\text{mean}(\\{r_j\\}_{j=1}^G) (setting aside the division by std for now). Subtracting a baseline is the standard variance-reduction technique for policy gradients, and the validity condition the textbook (RLHF Book, Chapter 6) gives is that the baseline must not depend on the current action, because \\mathbb{E}_{a\\sim\\pi}[b(s)\\nabla_\\theta\\log\\pi_\\theta(a|s)] = 0 holds for any b(s) that depends only on the state — the baseline leaves the expected gradient unchanged and only suppresses variance.\nThere is a detail here that few people mention: GRPO’s mean includes the very response being evaluated, so strictly speaking the baseline depends on the action, and the unbiasedness condition is not strictly satisfied. Compare RLOO (REINFORCE Leave-One-Out, same chapter of rlhf-book): its baseline is the mean of the other K-1 responses, deliberately excluding the response itself, to “keep the gradient estimator fully unbiased.” The relationship between the two advantages can be written out directly:\n\\hat{A}_i^{\\text{RLOO}} = r_i - \\frac{1}{G-1}\\sum_{j\\neq i}r_j = \\frac{G}{G-1}\\left(r_i - \\frac{1}{G}\\sum_{j}r_j\\right) = \\frac{G}{G-1}\\hat{A}_i^{\\text{GRPO(no std)}}\nThe directions are identical; the two differ only by a scale factor of G/(G-1). At G=64 the factor is 1.016, which hardly matters; at G=8 it is 1.14; at G=2 it is 2 — the GRPO-form advantage is only half as large as the leave-one-out form. This bias is not directional, just a rescaling that grows as G shrinks, and in practice its effect can be absorbed by the learning rate (this is an inference, not an experimental finding from any paper), but it explains why every property of small G gets amplified.\nThe second entry in the ledger is the baseline’s estimation noise. The group mean is an estimate of the true expectation \\mathbb{E}[r] from G samples, with noise of magnitude roughly \\sigma_r/\\sqrt{G} (\\sigma_r is the standard deviation of a single response’s reward). When G is small, this noise leaks straight into every response’s advantage: a correct response can end up with a negative advantage because “everyone else in its group happened to be strong too.” The probability that baseline noise randomly flips the gradient direction rises as G decreases — this is the standard “large G reduces variance” argument, and also what 2510.00977 sets out to challenge; more on that later.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"Role One: The Baseline's Sample Size","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"role-two-coverage-of-groups-with-effective-gradients","content":"Rewards in RLVR are binary (right or wrong; DAPO uses +1/−1, DeepSeekMath uses 1/0, and the two are equivalent). If a group’s G responses are all right or all wrong, there is no reward variance within the group and every advantage is zero — those G rollouts were run for nothing and contribute no gradient at all.\nThe probability that a group is uniform can be computed directly (what follows is my own calculation): let the model’s per-sample accuracy on the question be p, with samples independent; then\nP(\\text{zero-gradient group}) = p^G + (1-p)^G\nPlugging in numbers to see the trend (p is the per-sample accuracy; the table gives the probability that the group has zero gradient):\np G=8 G=64 0.5 0.8% ≈0 0.8 16.8% ≈0 0.9 43.0% 0.1% 0.95 66.3% 3.8% 0.99 92.3% 52.6% How to read it: for medium-difficulty questions (p in the 0.3–0.7 range), even a small G is basically safe; what really consumes G is questions at the difficulty boundary — at p=0.95, with G=8 two-thirds of the groups are all-correct and gradient-free, and only G=64 pushes that down to 4%. And the training trajectory pushes questions precisely toward the boundaries: for questions the model has learned, p \\to 1; for questions it can’t learn, p \\to 0; and the later in training, the fewer questions remain in the “gradient-bearing range.” This is one source of the “training efficiency and stability” problems that the DAPO paper describes in its motivation.\nFaced with the same table, there are two remedies. One is to increase G and brute-force p^G down — with costs growing multiplicatively. The other is DAPO’s Dynamic Sampling: instead of increasing G, simply rule that zero-gradient groups are not accepted. DAPO’s objective includes a hard constraint (the s.t. condition of Equation 8 in the paper):\n0 \u003c \\left|\\{o_i \\mid \\texttt{is\\_equivalent}(a, o_i)\\}\\right| \u003c G\nThat is, the number of correct responses in a group must lie strictly between 0 and G — all-correct and all-wrong groups are filtered out and replaced through oversampling, guaranteeing that every prompt in the batch has an effective gradient. For the same effective-group coverage, the price of filtering is additive oversampling rather than a multiplicative increase in G.\nBut there is another layer of coupling between filtering and G that is easy to miss: the filtering decision is itself based on an estimate from G samples. At G=4, “all 4 wrong” does not mean the model can’t solve the question — a question with p=0.3 has a 24% chance of getting all 4 wrong and being mistakenly thrown on the scrap heap. Pairing a small G with dynamic sampling will misjudge a whole set of medium-difficulty questions that “actually carry gradient value” as dead groups; the misjudgment rate falls exponentially with G ((1-p)^G, the same table). So dynamic sampling does not eliminate G’s coverage problem; it merely converts it from “zero gradient” into “mistaken discards” (this point is an inference; the DAPO paper does not discuss the misjudgment rate).","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"Role Two: Coverage of Groups With Effective Gradients","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"role-three-the-quality-of-the-std-normalization-denominator","content":"GRPO’s advantage is also divided by the group std: \\hat{A}_i = (r_i - \\text{mean})/\\text{std}. This std is likewise an estimate from G samples and is very noisy when G is small; for a uniform group the std is exactly 0, so implementations either add an eps or skip the group (setting a divide-by-zero trap).\nDr.GRPO’s critique of the std (see the dedicated post) — that when the std of all-correct or all-wrong groups approaches zero, the advantage is inflated to unreasonable magnitudes — is at its worst with small G: when G is small, the std is more likely to land on a small value, and extreme advantages show up frequently. Extreme advantages in turn push up how far the ratio moves, indirectly raising the clip fraction (this chain is an inference: abnormally large \\hat{A} → \\pi_\\theta moves more within a single step → w_t hits the clipping bound sooner). In other words, part of the claim that “large G stabilizes training” is really not about the baseline but about the std — two ledgers that the literature often lumps together.\nOne more theoretical result worth knowing (arXiv:2502.18548, “What is the Alignment Objective of GRPO?”): the authors give a framework characterizing GRPO’s stationary policies and point out that GRPO’s way of aggregating preferences differs fundamentally from the logarithmic pooling of standard RLHF — G affects not only the variance of the estimate but also the fixed point of the normalized objective itself. This is analytical work, and the conditions under which its conclusions hold (stationarity, infinitely many steps) are some distance from engineering practice, but it is a reminder of one thing: understanding G purely as a “sampling budget” is incomplete.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"Role Three: The Quality of the std-Normalization Denominator","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"counterpoint-are-2-responses-enough","content":"The argument of 2510.00977 is worth restating on its own terms: GRPO’s effectiveness comes from an implicit contrastive objective — comparing just two responses already yields a learning signal, and the group mean is merely a control variate for variance reduction that does not need to be a precise statistical estimate, so 2-GRPO with G=2 also works and, structurally, even ends up connecting to DPO. Their theoretical analysis and experiments (at small to medium scale) both support this conclusion.\nDoes this contradict DeepSeekMath’s G=64? Not necessarily. Of the things a large G buys, baseline precision is only one, and possibly the least important. The others are coverage of boundary-difficulty questions (the second role), the reliability of the filtering decision (the companion to dynamic sampling), and exploration coverage — a group needs at least one correct response for the model to have a “good sample” to learn from; for a hard question with p=0.1, there is an 81% chance at G=2 that the group contains no positive sample, versus only 0.1% at G=64. These gains depend strongly on model scale and on the task difficulty distribution: for a small model grinding through GSM8K-level problems, the p distribution is concentrated in the middle range, so a small G costs little; for a large model attacking AIME-level problems, p is pinned around 0.1, every positive sample is precious, and G is the exploration budget itself. Whether the 2-GRPO conclusion extends to large models in long-CoT settings is something nobody can answer yet (the paper’s experiments are limited in scale; this is a boundary, not a criticism).","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"Counterpoint: Are 2 Responses Enough?","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"the-cost-ledger-and-my-take","content":"First, the accounting: rollouts per training step = number of prompts × G. DeepSeekMath’s configuration (batch 1024, G=64) gets through only 16 prompts per step — prompt diversity and response diversity are traded against each other on the same budget sheet. One more configuration detail deserves attention: DeepSeekMath performs only one policy update per exploration stage (strictly on-policy, §4.2), so the ratio is always 1 and clipping never triggers, and G’s entire effect shows up in the baseline and in coverage; whereas with DAPO/GSPO-style multi-mini-batch updates, G also indirectly affects the clip fraction through std noise. The same G does not have the same marginal return in the two training regimes.\nMy take: there is no universally optimal G, because it spends money on three budget sheets at once — baseline precision (\\sigma_r/\\sqrt{G}, with the fastest-diminishing returns), effective-group coverage (p^G, with the most durable returns at boundary difficulty), and the exploration budget (the probability that a group contains a positive sample, almost linear on hard questions). The practical default (8–16) is the balance point under “a medium-difficulty distribution + dynamic sampling as a safety net”; 64 was the naive choice of the DeepSeekMath era, before dynamic sampling and before any clip modifications; 2 is the theoretical limit from the “contrastive objective” perspective, which in engineering terms nobody has yet dared to use at production scale. Before picking G, first ask yourself which of the three sheets is the bottleneck: if the questions are too hard (the exploration budget is tight), increase G or add dynamic sampling; if the questions are too varied (prompt diversity is tight), give the budget to the number of prompts; if training is unstable, check the std and the clip fraction first, and don’t rush to increase G.\nThe first four posts in this series happen to form a single thread: Clip-Higher fixes the token-level clipping bounds, GSPO changes the unit of the ratio, TIS plugs the numerical gap between training and inference, and this post goes back to the source of the signal — how to allocate the sampling budget. Every “self-evident” default configuration in GRPO, once taken apart, turns out to have a ledger of its own.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"The Cost Ledger, and My Take","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"open-questions","content":"There is no public systematic ablation of the optimal coupling between G, the learning rate, and the clip \\varepsilon; whether small G + a large learning rate can match the signal-to-noise ratio of a large G is backed only by scattered folk wisdom; No paper has quantified the actual impact on data efficiency of dynamic sampling’s “mistaken discard” rate ((1-p)^G) on medium-difficulty questions; Whether 2-GRPO holds for large models, long CoT, and boundary-difficulty tasks is one of the most worthwhile public replications to attempt right now.","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"Open Questions","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"references","content":"DeepSeekMath / original GRPO paper: arXiv:2402.03300; G=64, batch 1024, and the single-step on-policy update are in §4.2 DAPO paper: arXiv:2503.14476; the Dynamic Sampling constraint is in Equation 8, the ±1 reward in Equation 7 It Takes Two: Your GRPO Is Secretly DPO: arXiv:2510.00977 What is the Alignment Objective of GRPO?: arXiv:2502.18548 RLHF Book, Chapter 6 (the baseline unbiasedness condition, the RLOO leave-one-out formula): book/chapters/06-policy-gradients.md; teaching-code default num_rollouts: 8: code/policy_gradients/configs/ The zero-gradient-group probability table, the mistaken discards, and the clip-fraction chain are this post’s own calculations/inferences, each flagged where it appears in the text Previous post: The Ratio’s Wrong Unit: GSPO Goes Sequence-Level","date":"2026-08-29","permalink":"https://njx-njx.github.io/en/posts/grpo-group-size-tradeoffs/","section":"References","summary":"64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.","tags":["rl","post-training","grpo","rlvr","dapo"],"title":"Samples per Prompt: What GRPO's Group Size Buys"},{"anchor":"","content":"TRL’s SFTTrainer can train in a single line, so why does anyone still hand-write an SFT loop? I put every Stage 1 decision in the forge repo on the table one by one (chat template, prompt masking, collate and loss, bf16, LR schedule, data, validation) and check each against two textbooks (rlhf-book, hands-on-modern-rl), four production frameworks (alignment-handbook, trl, open-instruct, OpenRLHF), and six information sources: which decisions have consensus, which are genuine recipe disputes, and which the two textbooks each gloss over.\nThis post comes with a companion interactive report (in Chinese), which includes a decision comparison matrix and a drill-down index of sources.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Anatomy of a Minimal SFT Trainer: Every Decision","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"why-hand-write-a-trainer-of-only-511-lines","content":"I’m building forge, a post-training repo that grows stage by stage along my learning path (Stage 1: SFT; local, not yet pushed as open source). It has only one principle: use mature engineering components off the shelf, but write every line of the core training-data path myself and be able to explain why.\nThere are three routes to writing SFT. The hands-on-modern-rl course’s sft_pipeline.py is a one-line call to TRL’s SFTTrainer, with the training loop, loss, and padding all inside the library; the instruction_tuning code from Nathan Lambert’s “RLHF Book” is a hand-written loop; production frameworks (alignment-handbook, open-instruct, OpenRLHF) stack a whole suite of packing, flash-attention, and decontamination on top of the Trainer.\nforge went with hand-writing. The motivation isn’t performance—a 0.5B model trains fast no matter how you train it—but that every decision must be explicitly visible, because every later stage (RM, DPO, GRPO, agentic RL) will reuse this loop, and the RL stages will also have to stuff sampling and reward into it; the black box has to be opened sooner or later. This post lines up each of forge’s decisions against two textbooks, four production frameworks, and six information sources (Notion notes, Zhihu, a Feishu industry-news group, arXiv, local video courses). Conclusion first: most decisions have strong consensus, a few are genuine recipe disputes, and then there are a few pitfalls that the two textbooks each paper over in different ways.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Why Hand-Write a Trainer of Only 511 Lines","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"decision-1-chat-templatethree-pitfalls-in-string-concatenation","content":"The raw data is structured messages (role + content), but the model only understands plain text, so the first step is rendering. forge hand-writes ChatML: each message is wrapped as \u003c|im_start|\u003e{role}\\n{content}\u003c|im_end|\u003e\\n. I verified that this is character-for-character identical to the output of the chat_template bundled with Qwen’s official tokenizer.\nIt looks like there’s no real decision in this, but three sources each turned up a pitfall:\nBase models have no template. OLMo-2-1B is a base model, and its tokenizer has no chat template in it; rlhf-book’s approach is to copy the entire chat_template attribute from the tokenizer of the sibling model allenai/OLMo-2-0425-1B-SFT (utils.py:82-88). In the real world, a template isn’t necessarily ready-made. apply_chat_template has hidden behavior. When the messages have no system turn, Qwen’s template injects a default system prompt (“You are Qwen, created by Alibaba Cloud…”). The hands-on course’s data has no system turn, so its training text actually carries this default system line—using the tokenizer’s bundled template can bring in text you never reviewed. forge renders only the turns already present in the sample and injects nothing. Template errors are SFT’s number-one silent problem. HuggingFace’s official blog published “Chat Templates: An End to the Silent Performance Killer” (2024-01; my Notion materials library tags it as “the most frequent SFT pitfall”); and the first item on the Chinese community’s Trainer troubleshooting checklist is “loss not going down → wrong chat template / wrong labels mask”. My trade-off: hand-write the template first to understand its structure (a template is just string concatenation; the special tokens are registered in the vocabulary, so \u003c|im_start|\u003e encodes to exactly one whole token and doesn’t fragment), so that later, when I hook into the official template, I know what I’m hooking into.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Decision 1: Chat Template—Three Pitfalls in String Concatenation","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"decision-2-prompt-maskingthe-biggest-point-of-contention-in-this-post","content":"For the rendered sequence, every position needs a decision on whether it counts toward the loss. The rule is: the logits at each position predict the next token; if the prediction target is an assistant token, it is scored, otherwise it is set to -100. forge’s choice is to count every assistant turn (including the wrapping markers) and mask all user/system turns.\nHow this decision is distributed across the sources is interesting:\nrlhf-book trains only on the final assistant turn (assistant replies in the middle of a multi-turn conversation are masked too), and even masks the \u003c|assistant|\u003e generation header itself (utils.py:138,142-148). Chapter 4 of its textbook explicitly lays out both conventions side by side: train only on the final turn, or mask only the user turns. forge chose the latter. The hands-on course’s code doesn’t mask at all: it feeds SFTTrainer a “text” column and computes the loss over the entire sequence. Yet the course’s own docs (imitation-learning-pipeline.md:236-238) say that counting user/system tokens in the loss is “the most common pitfall… this isn’t pedantry; it’s protecting the model you train from reciting the prompts”. Same repo, and the code and the docs contradict each other—the most direct example of “teaching implementation ≠ best practice”. trl’s current approach is assistant_only_loss + {% generation %} markers in the template, with an explicit warning: if an assistant turn’s end token isn’t inside the loss mask, “the model may never learn to stop”. This corroborates the need for forge to train on \u003c|im_end|\u003e—the model must learn to stop. Ownership of the boundary tokens: production templates (trl/OpenRLHF) usually wrap only the assistant content in the generation markers, i.e., the \u003c|im_start|\u003eassistant\\n header doesn’t count toward the loss; forge counts the header too. A difference of a token or two—a detail open to debate. Masking itself has been challenged by research: “Instruction Tuning with Loss over Instructions” (arXiv 2405.14394) found that also computing the loss on the instruction part (IM) outperforms computing it only on the output (OM) in many settings across 21 benchmarks. In other words, completion-only is the current framework default, not something proven correct—when the goal is for the model to internalize the instruction distribution, masking throws away signal. Written out as a formula, forge’s loss is:\nL = -\\frac{1}{|A|}\\sum_{t:\\,x_{t+1}\\in A}\\log p_\\theta(x_{t+1}\\mid x_{\\le t})\nwhere A is the set of tokens in assistant turns and |A| is the number of valid tokens.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Decision 2: Prompt Masking—The Biggest Point of Contention in This Post","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"decision-3-collate-and-lossline-for-line-isomorphic-plus-a-tail-neither-book-handles-well","content":"The core trio on the training side: right-pad collate (pad tokens fill the inputs, -100 fills the labels, attention_mask is marked 0), shift (logits[:, :-1] against labels[:, 1:]), and F.cross_entropy(ignore_index=-100). This implementation is line-for-line isomorphic to rlhf-book’s—two independently written pieces of code that look the same are the strongest evidence that “this is the standard practice”.\nBut the side-by-side comparison exposed two problems that not even the textbooks handle cleanly:\nThe grad accumulation tail. The number of micro-batches in an epoch isn’t necessarily divisible by the number of accumulation steps. rlhf-book simply discards the leftover group at the tail—the backward passes are wasted, and those few samples contribute nothing to the update; forge steps on the leftover group as well, but still divides the loss by the nominal accum count—so the leftover group’s gradient is applied at a scale that’s too small. Each of the two books fudges half of it. What’s more common in real production is to normalize by the leftover group’s actual size, or simply set drop_last=True and align the scheduler’s step count. The micro-batch averaging convention. Both forge and rlhf-book “average within each micro-batch first, then divide by accum”. When samples differ greatly in their number of trainable tokens, this differs subtly from “a weighted average over all tokens”. The vast majority of frameworks accept this compromise, but if you want to pick at training details (mixed data with large length differences), this is worth coming back to. The production-side counterpart is trl’s chunked_nll: drop the -100 positions first, then compute the projection—mathematically equivalent, and it saves peak GPU memory. OpenRLHF has another detail worth copying: sequences are forced to end with EOS—after truncation, input_ids[-1] = eos_token_id (sft_dataset.py). forge makes no such guarantee: the last token of a truncated sample is arbitrary, and when truncation happens to cut off the \u003c|im_end|\u003e at the end of an assistant turn, that amounts to teaching the model to “stop halfway through”.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Decision 3: Collate and Loss—Line-for-Line Isomorphic, Plus a Tail Neither Book Handles Well","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"decision-4-precisionpure-bf16-and-a-redundant-layer-of-autocast","content":"Both textbooks load the weights in pure bf16: no autocast, no fp32 master weights, no GradScaler. Teaching code goes for convenience, and a 0.5B model really doesn’t call for more anyway.\nforge’s approach exposes something I need to correct: the weights are already loaded in bf16 (model.py), yet the forward pass is additionally wrapped in a torch.autocast(bf16) (train.py:92)—when the weights are already bf16, this autocast layer is essentially redundant. Real mixed precision (fp32 master + loss scaling) lives at the Megatron/DeepSpeed level; Stage 1 doesn’t need it, but it deserves a note in the blog. Tulu 3’s --mixed_precision bf16 and the community’s “BF16 is 2x faster than FP32” (hands-on course docs) give this default production-side backing.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Decision 4: Precision—Pure bf16 and a Redundant Layer of Autocast","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"decision-5-lr-schedulecosine-vs-linear-is-a-recipe-dispute","content":"forge uses get_cosine_schedule_with_warmup with 3% warmup. Production gives no single answer: Tulu 3’s 8B SFT recipe is lr 5e-6 + linear decay + warmup 0.03 (open-instruct finetune_8b.sh; a reproducible config is harder evidence than a paraphrase of the paper); Zephyr uses 2e-5 + cosine + warmup 0.1; OpenRLHF defaults to cosine_with_min_lr. rlhf-book simply hand-writes a linear decay—the teaching book itself doesn’t use cosine either.\nTwo worthwhile observations:\nThe default scheduler is a trap. If you don’t specify one explicitly, HF/TRL default to linear + warmup 0 (that’s exactly how the hands-on course’s code was run). You think you didn’t make a decision, but you already did. lr and packing are strongly coupled. The OLMo 3 practice cited in the textbook: sequence packing significantly increases the effective token batch, so the lr has to go up with it. Talking about lr in isolation from the throughput configuration is meaningless. forge’s numbers match mainstream recipes item by item: lr 1e-5 (within the 5e-6~2e-5 production range, and in line with “the smaller the model, the larger the lr”), warmup 0.03, weight decay 0.0, grad clip 1.0, epochs 2. An effective batch of 32 (8×4) is teaching-scale for 0.5B—rlhf-book coincidentally also uses 32 (4×8), while production uses 128-256. epochs=2 has research-side support: SFT scaling research shows that the gains from repeated data come roughly within the first 4 epochs (arXiv 2402.17193), and “SFT Memorizes, RL Generalizes” (arXiv 2501.17161) further points out that training for more epochs mainly reinforces memorization rather than generalization—2 epochs is in the safe zone, and it also foreshadows the ceiling of a pure SFT slice.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Decision 5: LR Schedule—Cosine vs. Linear Is a Recipe Dispute","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"decision-6-dataforges-biggest-simplification-reveals-the-full-production-picture","content":"forge uses 10,000 examples from the everyday-conversations subset of HuggingFaceTB/smoltalk: a single source, zero quality filtering. This is the biggest simplification in the entire repo, and the sources pin down its place precisely:\nThe SmolTalk dataset card (verified) states: the full set has 1.1M examples in 14 subsets, and the core new data is Smol-Magpie-Ultra 400K (Magpie pipeline + Llama-3.1-405B generation); everyday-conversations is multi-turn everyday dialogue, merely a “basic conversational behavior” slice, with math, code, and long context covered by other subsets. In other words, training on it alone yields the conversational layer of the full recipe, not full capability.\nBoth ends of the data-scale spectrum have names: LIMA (arXiv 2305.11206) used 1,000 hand-curated examples to bring a 65B model close to the GPT-4 of its day and proposed the “Superficial Alignment Hypothesis”—pretraining has already stored up the capabilities, and SFT only does shallow alignment; Tulu 3 (arXiv 2411.15124) used 940K examples for rigorous multi-skill mixture ablations, backed by a decontamination toolchain. In between sit AlpaGasus (90% of the 52K was low quality; the 9K filtered out of it beat the full set) and Deita (scoring on complexity, quality, and diversity; 6-10K beat larger mixed sets). One line in my Notion notes sets the tone for this spectrum: LIMA coexists with industrial million-scale datasets, and every hard capability needs data behind it; data work comes down to just four decisions—what to get, what to keep, how much to mix in, and what to do when it isn’t enough.\nAs for what production data engineering actually looks like, the Chinese community and the Feishu group supplied the two ends: a Zhihu column calls the curation pipeline (dedup → quality gate → remove overlap with eval sets → ensure multi-turn completeness) “the link in the whole pipeline that deletes the most and is the least transparent”; on mixture ratios, mixing in 10%-20% general data to prevent catastrophic forgetting is the highest-consensus rule of thumb in the community (there are also two claims a factor of ten apart, 1:1 and 1:10~1:100 experience replay—the ratio itself is unsettled). Claims like “90% of SFT performance depends on data quality” circulate widely in the community, but no experimental source can be given for them, so they can only be treated as a sentiment indicator.\nNewer signals come from the industry-news group: Shanghai AI Lab’s SKT, AutoCompact’s judge-guided SFT, MindForge’s trajectory fine-tuning—production SFT data is increasingly “strong-model trajectories + verifiers”, and the smoltalk that forge loads directly is, in essence, also trajectories that someone else has already distilled. Another paper shared in the group, “SFT Conflicts, RL Coexists” (arXiv 2608.03573), supplies a footnote from the opposite direction: multi-stage multi-task SFT degrades by 23.1% on average, while the same multi-stage RL improves by 24.9%—the pitfall a minimal single-task, single-stage SFT sidesteps isn’t in the loop itself but in the data composition.\nMy judgment: 10,000 examples from a single source is a reasonable teaching starting point—first verify that the pipeline is correct; mixtures and quality gates are Stage 2’s business. One practical constraint to keep firmly in mind: right now max_len=1024, while long reasoning-chain data routinely runs 2000-8000 tokens, so when switching data the context length has to change along with it; otherwise a large number of samples will be truncated into “the question plus half a thinking process”, with \u003c|im_end|\u003e cut off in the bargain.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Decision 6: Data—forge's Biggest Simplification Reveals the Full Production Picture","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"decision-7-validationval-loss-is-the-low-variance-choice-but-only-half-the-story","content":"forge’s validation trio: hold out 1% of the data as a val split, compute val loss after training, and greedy-decode 8 fixed prompts to eyeball the results.\nA methodological rule in my Notion notes directly backs val loss: accuracy-type metrics have high variance on small models, so prefer per-domain validation loss, repeated across multiple seeds. Run an MMLU-style eval on a 0.5B model and the noise will drown out the signal; loss is at least stable.\nThe course docs (unreviewed, but this point matches frontline experience) list five metrics to watch in SFT: train_loss, eval_loss, format pass rate, repetition rate, and manual spot checks, and warn that “looking only at loss is a beginner’s mistake—the loss will look great, but the model will turn into an assistant with correct formatting and anemic content”. rlhf-book’s approach supplies the other half: during training it periodically generates samples from 4 fixed prompts, and at step 0 it first prints the base model’s outputs—the qualitative contrast of “base misses the question → after SFT it answers and then stops” is the most intuitive way to check whether a checkpoint is taking effect. Ironically, the entire module never saves the model (verified with grep): it has a tracker and in-training generation, but no checkpoint.\nWhat forge currently lacks: periodic generation during training, quality metrics like format pass rate, and a wandb tracker. Plus a set of throughput optimizations from production practice: packing + padding-free + flash-attention (Netflix’s write-up reports 4.7x throughput, and IBM’s arXiv 2407.09105 is the authoritative account of putting it into production), length-bucketed batching, and gradient checkpointing on the training side. None of these go into Stage 1—the point of minimalism is to let you see clearly what the system is doing without them.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"Decision 7: Validation—Val Loss Is the Low-Variance Choice, but Only Half the Story","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"the-not-done-list-is-the-stage-2-roadmap","content":"Having compared against all six sources, here’s what forge deliberately left out and what it should add soon, in order of priority:\nP0: packing + padding-free + FA2. All four production frameworks have it, and it’s the number-one engineering gap between forge and production; Netflix’s 4.7x throughput shows this is no mere detail. P1: align the mask boundaries with the production convention (assistant header excluded from the loss, end token always included), and force truncated samples to end with EOS (a one-line fix, as in OpenRLHF). P1: normalize the accum tail by its actual group size; and while at it, remove that redundant autocast layer. P2: chunked loss, grad checkpointing, decontamination—these can wait for the large-model stage; decontamination belongs to the data-preparation layer and doesn’t go into the trainer. Not pursuing: NEFTune (frameworks support it, mainstream recipes don’t use it), switching cosine to linear (a recipe dispute with two coexisting camps; forge sides with the majority). Looking back, what hand-writing 511 lines bought me isn’t performance but a decision checklist: why each knob sits where it does, how the textbooks choose, how production chooses, and what research says. SFTTrainer runs in one line, but that one line hides a dozen-plus default decisions, three of which (full-sequence loss, linear schedule with zero warmup, silent truncation) happen to be among the top entries on the community’s troubleshooting checklists. Only by taking the defaults apart do you know what you’re choosing.","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"The Not-Done List Is the Stage 2 Roadmap","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"references-and-sources","content":"Textbook code: rlhf-book code/instruction_tuning/ (repo; key line numbers are given in the main text); hands-on-modern-rl code/chapter08_rlhf/sft_pipeline.py and docs/chapter15_rlhf/ (repo; the course states that it has not been comprehensively reviewed).\nProduction frameworks (all main-branch source snapshots): alignment-handbook recipes/zephyr-7b-beta/sft/config_full.yaml; trl sft_trainer.py/sft_config.py; open-instruct finetune.py and scripts/train/tulu3/finetune_8b.sh; OpenRLHF openrlhf/datasets/sft_dataset.py.\nPapers: LIMA 2305.11206 · AlpaGasus 2307.08701 · Deita 2312.15685 · Tulu 3 2411.15124 · SmolLM2 2502.02737 · Magpie 2406.08464 · SFT scaling 2402.17193 · SFT Memorizes, RL Generalizes 2501.17161 · Instruction Modelling 2405.14394 · Packing with FlashAttention 2407.09105 · SFT Conflicts, RL Coexists 2608.03573.\nCommunity and industry: HuggingFace · Chat Templates: An End to the Silent Performance Killer · Zhihu · SFT data curation · Zhihu · Training data preparation for SFT and RL · Netflix engineering blog · Scaling LLM Post-Training (indexed via the Notion materials library, 2026-02) · Feishu industry-news group (posts on SKT, AutoCompact, MindForge, 2026-05 to 2026-08).\nforge repo: src/forge/data.py, src/forge/train.py, src/forge/model.py, configs/sft.yaml, tests/test_data.py (local, not yet pushed as open source; line numbers in the text refer to the 2026-08-28 working tree).","date":"2026-08-28","permalink":"https://njx-njx.github.io/en/posts/forge-sft-anatomy/","section":"References and Sources","summary":"Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.","tags":["sft","post-training","forge","engineering","llm"],"title":"Anatomy of a Minimal SFT Trainer: Every Decision"},{"anchor":"","content":"Rollouts run on an inference engine and training runs on a training engine, and the same model’s numbers don’t match across the two: the distribution you actually sample from deviates from the policy you assume, and nominally on-policy training quietly becomes off-policy. This post breaks down where the discrepancy comes from, its two consequences (biased gradients and a deployment gap), the lineage of algorithmic patches, and Sea AI Lab’s conclusion tracing the root cause to BF16 precision.\nThis post has a companion interactive report (in Chinese), with a drill-down diagram of the two-engine mechanism, a comparison of the patch lineage, and a bit-field breakdown of the precision formats.","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"One Model, Two Probabilities: Cost of Mismatch","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"the-same-hard-to-explain-collapse","content":"In the second half of 2025, several teams independently reported the same phenomenon: early in GRPO training everything looks normal and reward climbs steadily; then, at some point, it abruptly turns downward until training collapses. There is no gradient spike before the collapse, nothing is wrong with the hyperparameters, and if you rerun with a different batch of training queries, the collapse still comes, just at a different time.\nThe August 2025 technical report by Defog (Feng Yao, ByteDance Seed), “Your Efficient RL Framework Secretly Brings You Off-Policy RL Training”, offered an explanation that surprised quite a few people: you think you are doing on-policy training, but the distribution your rollout data actually comes from has already drifted away from the policy you are optimizing. The problem is not in the algorithm but in the numerics—the probabilities computed by the engine that generates the data don’t match those computed by the engine that trains on it.","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"The Same Hard-to-Explain Collapse","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"why-one-model-computes-two-probabilities","content":"Modern LLM RL frameworks are split into two halves. The rollout side uses an inference engine (vLLM, SGLang), heavily optimized at the kernel level for high throughput; the training side uses a training engine (Megatron, PyTorch FSDP, DeepSpeed), which computes each token’s log probability and the gradients. Both sides load the same weights, so in theory they should be the same policy \\pi.\nBut because their engineering implementations differ, the engines don’t compute exactly the same results. Sea AI Lab’s FP16 paper (arXiv:2510.26788) summarizes where the discrepancies come from:\nDifferent computation patterns. Inference generates autoregressively, token by token; training runs a parallel forward pass over the whole sequence under teacher forcing. For the same token string, the different computation processes use different floating-point reduction orders, and errors accumulate differently. Different kernel implementations. The inference engine’s RMSNorm, RoPE, and attention kernels are specially tuned for latency and throughput, and differ numerically, by tiny amounts, from the training-side implementations. Different parallel sharding. Training and inference use different tensor-parallel and data-parallel sharding, and a different order of cross-GPU reductions changes the floating-point result. MoE top-k gating is sensitive to small changes. Expert selection is sensitive to tiny fluctuations in the logits, so if the same weights are run once through each of the two engines, the selected experts can differ. The 10% expert drift that the GSPO post mentions occurs along the temporal dimension of “before vs. after an update”; here we mean the spatial dimension of “same weights, two engines”. Gating sensitivity introduces a discrepancy along each of the two dimensions separately. The precision format’s mantissa is too short. The default BF16 has only 7 mantissa bits, so none of the tiny discrepancies cited above has enough numerical precision headroom, and they end up producing noticeable deviations. This point is the crux of the whole post; more on it later. So the inference engine actually corresponds to a numerically different policy from the training engine’s, denoted \\mu. Data is sampled from \\mu, but the gradient is computed under the assumption of \\pi. The boundary between on-policy and off-policy ultimately comes down to the floating-point behavior of kernels.","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"Why One Model Computes Two Probabilities","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"two-consequences-biased-gradients-discounted-deployment","content":"Consequence one: biased gradients. The theoretical policy gradient requires sampling from the policy being optimized, \\pi:\n\\nabla_\\theta\\mathcal{J}(x,\\theta)=\\mathbb{E}_{y\\sim\\pi(\\cdot|x,\\theta)}\\left[\\nabla_\\theta\\log\\pi(y|x,\\theta)\\cdot R(x,y)\\right]\n\\pi(y|x,\\theta) is the likelihood of the response in the training forward pass, R(x,y) is the reward, and the expectation has to be taken under the distribution of \\pi. But the actual samples come from \\mu:\n\\nabla_\\theta\\mathcal{J}_{biased}(x,\\theta)=\\mathbb{E}_{y\\sim\\mu(\\cdot|x,\\theta)}\\left[\\nabla_\\theta\\log\\pi(y|x,\\theta)\\cdot R(x,y)\\right]\\neq \\nabla_\\theta\\mathcal{J}(x,\\theta)\nThe distribution underlying the expectation has suddenly changed, and the equality no longer holds. The bias is usually small, but RL is an iterative process that keeps pushing the distribution in one direction, so small differences between \\mu and \\pi get amplified step by step over training. The paper also observes a phenomenon that can serve as an early-warning signal: in every run that eventually collapsed, the gap between \\pi and \\mu kept widening before the collapse; in extreme cases, under the same weights, the same token had a probability approaching 1 in one engine and approaching 0 in the other.\nConsequence two: the deployment gap. Even if the gradient bias in training is fully corrected, the optimization target is still the training engine’s probability distribution, while deployment and evaluation run on the inference engine:\n\\arg\\max_\\theta \\mathbb{E}_{y\\sim\\mu}R \\neq \\arg\\max_\\theta \\mathbb{E}_{y\\sim\\pi}R\nParameters that are optimal under \\pi are not necessarily optimal under \\mu. This gap is visible in the paper’s experiments: the most stable algorithmic patch under BF16 (sequence-level masked correction) reaches a peak training accuracy of 95% and an AIME 2024 score of 34%; under FP16, the numbers are 99% and 39%. On this measure, all the algorithmic patches make zero difference—they only address the bias in the training process and cannot address this deployment gap.","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"Two Consequences: Biased Gradients, Discounted Deployment","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"algorithmic-patches-and-their-limits","content":"At its core, the problem is “sampling from \\mu but taking the expectation under \\pi”, and the textbook fix is an importance-sampling correction: multiply each sample by the train-inference probability ratio \\pi/\\mu. The improvements in the second half of 2025 came mainly in two steps:\nPatch one: token-level truncated correction (Defog). For each token, compute the ratio of the training probability to the inference probability, \\rho_t = \\pi(y_t|x,y_{\u003ct},\\theta')/\\mu(y_t|x,y_{\u003ct},\\theta'), truncate it at an upper bound C (3 in the experiments), and multiply it into each term of the GRPO gradient. Note that the denominator \\mu is the inference engine’s probability, so the training engine has to run one more forward pass over the same sample to obtain the numerator, adding about 25% training overhead (estimated assuming the backward pass costs roughly 2× the forward pass). Result: the collapse is delayed, but training still collapses after peaking at 82% (VeRL) / 88% (Oat).\nPatch two: sequence-level masked correction (“When Speed Kills Stability”, Liu et al.). The authors point out that the token-level correction yields a biased gradient, and change the correction to compute a single ratio for the entire sequence, \\rho = \\pi(y|x,\\theta')/\\mu(y|x,\\theta'), masking out the gradient of the whole sequence whenever it exceeds C. The idea is in the same lineage as GSPO (although GSPO’s s_i serves only as the quantity being optimized, whereas \\rho here is a correction factor). It is the only algorithmic patch that does not collapse under BF16; the price is high variance in the sequence-level ratio and slow convergence, and the deployment gap remains exactly as it was—the peak is only 95%/34%.\nWhat the two patches have in common: both accept the fact that \\mu \\neq \\pi and compensate sample by sample, at the cost of extra compute and added variance, and neither can solve the deployment gap.","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"Algorithmic Patches and Their Limits","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"the-root-cause-bf16s-7-bit-mantissa","content":"Rather than pushing further down the patching path, the Sea AI Lab paper went straight to examining the numerical format.\nBF16 and FP16 both use 16 bits but allocate them differently. BF16 uses 8 exponent bits + 7 mantissa bits, giving it a dynamic range on par with FP32: pretraining doesn’t have to worry about overflow and doesn’t need loss scaling, which is why it became the default. FP16 uses 5 exponent bits + 10 mantissa bits: its dynamic range is much smaller (it relies on loss scaling to prevent gradient underflow), but adjacent representable numbers are 8× as dense as in BF16 (2^{10} vs. 2^7), so its precision is higher.\nThe paper’s assessment: the requirements change in the RL fine-tuning stage. The dynamic range of weights and activations has already been settled during pretraining, so BF16’s FP32-level range no longer serves any practical purpose here, while every kernel discrepancy between training and inference needs FP16’s 3 extra mantissa bits to provide enough numerical precision headroom. Once precision is unified to FP16, the two engines’ outputs agree numerically on the vast majority of tokens, and \\mu and \\pi coincide again. Offline analysis shows that under BF16 the sequence-level train-inference mismatch grows exponentially with generation length, while FP16 reduces the mismatch substantially, to about 1/24 of BF16’s. In the precision ablations, the “BF16 training + FP32 inference” combination is also fully stable, but inference is 3× slower; using FP16 throughout is the best combination of stability and efficiency.\nThe most telling result is the next set of experiments: with FP16, the performance differences between algorithms almost disappear. GRPO, token-level TIS, GSPO, and sequence-level MIS, which differ markedly under BF16, end up close to one another under FP16, and even the plainest importance-sampling policy gradient, without any correction, achieves the best performance. The reason is not hard to see: once the gap between \\mu and \\pi is close to zero, training returns to an approximately on-policy regime, and every algorithm meant to “correct the mismatch” loses the thing it was correcting. (One more detail: on VeRL, GSPO’s gradients turn into NaN after 1,200 steps, which suggests that algorithm-level improvements may be of no use while the numerical problem remains unsolved.) The engineering routes cited here bear this out too: the SGLang team hand-aligned the kernel implementations of the two compute paths (the November 2025 Unified FP8 work), and Miles/slime aligned training and inference in one go at the system level. Both are effective, but they require deep kernel expertise and a heavy engineering investment, and they are hard to port. The numerical format is the cheapest of all the solutions: it only takes changing a few lines of configuration, while keeping the existing loss scaling mechanism.","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"The Root Cause: BF16's 7-Bit Mantissa","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"reading-it-alongside-the-previous-two-posts","content":"The three posts in this series happen to approach the same problem at four levels:\nDAPO Clip-Higher: assumes training and inference are consistent, and adjusts the clip bounds inside the algorithm to fix entropy collapse; GSPO: moves the optimization objective to sequence-level units, is naturally robust to the discrepancy in \\mu, and also tolerates MoE routing drift; TIS algorithmic patches (the first half of this post): accept the fact that \\mu \\neq \\pi and apply importance-sampling compensation sample by sample; FP16 and kernel alignment (the second half of this post): make \\mu \\approx \\pi, eliminating the mismatch directly. Seen against this arc, GSPO §5.4’s observation that “sequence-level likelihoods are highly tolerant of precision differences between training and inference” is no longer just a side benefit—it is a direct algorithm-level response to the train-inference mismatch (TIS) problem.","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"Reading It Alongside the Previous Two Posts","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"my-three-takes","content":"First, in engineering terms, on-policy is a numerical problem. The algorithm’s definition says “sample from the current policy”; in engineering practice you can only sample from “a numerical approximation of the current policy under some kernel implementation”. Once the two differ by a certain magnitude, the premises of the theoretical analysis no longer hold, and existing alerting mechanisms cannot raise an alarm about it: the training curve looks normal, the loss looks normal, and only the logprob gap between the training side and the inference side is silently diverging. The practical advice is straightforward: make the train-inference logprob gap (or KL) a routine monitoring metric. Its cost is close to zero, and the early-warning value the paper demonstrates is empirically measured.\nSecond, BF16’s default status deserves to be questioned, but the FP16 conclusion should not be over-generalized. The evidence covers 1.5B to 30B MoE, two frameworks, LoRA, and two model families, which is solid; the paper itself also states that it does not claim universality: using FP16 on very large models requires managing overflow, and if the industry moves toward FP8 (with only 3–4 mantissa bits), the train-inference mismatch problem will reappear. The current FP8 RL route (LMSYS, NVIDIA) unifies training and inference on FP8 rather than going back to FP16. The precision problem will either be taken seriously again or resurface in another form.\nThird, the deployment gap is the most direct and least studied problem behind all this. Although the gradient bias can be corrected, and the train-inference mismatch can be eliminated by aligning both the kernels and the precision format on the two sides, the problem that “the trained parameters are not optimal for the deployment engine” has not disappeared; it has only shrunk. When evaluation switches to the inference engine, which policy does the reported score actually belong to? With mixed multi-engine deployment, how should the gap be computed? So far I have not seen a systematic study.","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"My Three Takes","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"open-questions","content":"Whether the small train-inference discrepancies that remain under FP16 will accumulate back into a problem over ultra-long training runs of thousands of steps; The final performance of the unified-FP8 route versus the FP16 route under the same compute budget: there is currently no public head-to-head comparison; The deployment gap will still appear, only smaller: when evaluating with an engine different from the training environment, which policy should the score be attributed to?","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"Open Questions","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"references","content":"Sea AI Lab / NUS: Defeating the Training-Inference Mismatch via FP16 (the source of this post’s experiments, formulas, and data) Defog (Feng Yao et al.): “Your Efficient RL Framework Secretly Brings You Off-Policy RL Training”, 2025-08, published on Notion; the original report could not be accessed directly due to technical limitations, so its token-level correction formula is paraphrased from §2.1 of the FP16 paper Liu et al.: “When Speed Kills Stability: Demystifying RL Collapse from the Inference-Training Mismatch”, 2025, published on Notion; sequence-level masked correction, paraphrased in the same way Engineering routes: Unified FP8: Moving Beyond Mixed Precision for Stable and Accelerated MoE RL (LMSYS); Miles/slime: All-In-One Solution to Training-Inference Mismatch Textbook hands-on-modern-rl, Appendix B.1: two engineering treatments of the old logprobs produced by the rollout engine (use the results returned by the inference side directly, or recompute them on the training side and check their accuracy) Previous posts: Entropy Collapse: What DAPO’s Clip-Higher Fixes and The Ratio’s Wrong Unit: GSPO Goes Sequence-Level","date":"2026-08-27","permalink":"https://njx-njx.github.io/en/posts/tis-train-inference-mismatch/","section":"References","summary":"Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.","tags":["rl","post-training","grpo","tis","precision"],"title":"One Model, Two Probabilities: Cost of Mismatch"},{"anchor":"","content":"GRPO carries over PPO’s token-level importance ratio, which the Qwen team deems a fundamental misuse of importance sampling that leads to irreversible collapse in long-response MoE training. GSPO lifts the ratio, clipping, and optimization all to the sequence level: it clips two orders of magnitude more tokens, yet trains more efficiently. This post unpacks that chain of argument, along with the premises and costs the paper leaves unstated.\nThis post has a companion interactive report (in Chinese), with drill-down formula breakdowns, a clipping-band comparison, and a dilution-effect calculator.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"the-problem-a-patch-exposed","content":"Before mid-2025, training MoE models with GRPO required a dedicated patch.\nIn the GSPO paper (arXiv:2507.18071), the Qwen team describes the approach they had been using: Routing Replay. During sampling, the expert routes activated by the old policy \\pi_{\\theta_{old}} are cached; when computing the importance ratio, these routes are “replayed” on the new policy \\pi_\\theta, so that the ratio’s numerator and denominator run through the same activated network. The patch was needed because of one number they observed: in the 48-layer Qwen3-30B-A3B, after each gradient update, roughly 10% of the experts activated for the same rollout sample under the new policy differ from those under the old policy. Once the routing changes, the per-token probabilities change with it, the token-level importance ratios fluctuate violently, and GRPO simply fails to converge.\nThe patch works, but its costs are real: extra GPU memory and communication overhead, and forcing the new policy onto the old routes amounts to capping the model’s effective capacity. What is more troubling is something else — even with the patch, GRPO on long-response tasks with large models still suffers irreversible model collapse. The paper’s wording is strong: once collapse occurs, rolling back to a checkpoint, carefully tuning the clipping range, extending the generation length, switching RL queries — none of it can bring the model back.\nThe token-level importance ratio has been standard since the PPO era, in use for eight years. Why has it suddenly turned pathological in RL for LLMs? GSPO’s answer: it did not suddenly fall ill; it was used in the wrong place from the start — this is a misuse of importance sampling, and in the era of small models, short responses, and near-on-policy training, the error was simply masked.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"The Problem a Patch Exposed","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"why-importance-sampling-fails","content":"Let me first write down importance sampling (IS) itself. We want to use samples from a behavior distribution \\pi_{beh} to estimate the expectation, under a target distribution \\pi_{tar}, of a function f:\n\\mathbb{E}_{z\\sim\\pi_{tar}}[f(z)] = \\mathbb{E}_{z\\sim\\pi_{beh}}\\left[\\frac{\\pi_{tar}(z)}{\\pi_{beh}(z)} f(z)\\right]\nThis identity does not hold by virtue of the ratio itself; it relies on averaging over many samples: only when N \\gg 1 do the fluctuations of the random ratio w(z)=\\pi_{tar}(z)/\\pi_{beh}(z) average out, so that the estimate converges to the true value. Multiplying a single sample by w provides no correction whatsoever; it merely multiplies f(z) by a high-variance random number.\nNow look at what GRPO does. It computes a ratio for every token:\nw_{i,t}(\\theta)=\\frac{\\pi_\\theta(y_{i,t}|x,y_{i,\u003ct})}{\\pi_{\\theta_{old}}(y_{i,t}|x,y_{i,\u003ct})}\nHere y_{i,t} is the i-th response’s t-th token, and y_{i,\u003ct} is the context preceding it. \\pi_{\\theta_{old}}(\\cdot|x,y_{i,\u003ct}) is the next-token distribution at step t, and only one sample is drawn from it: y_{i,t}. In other words, every ratio uses a single sample to “correct” a single-step distribution — by the criterion above, this does not constitute an IS correction at all; it merely injects a per-token random weight into the gradient. The longer the sequence, the more random weights accumulate; the larger and sparser the model (MoE), the higher the variance of each individual weight. Clipping then zeroes out the gradients of tokens that fall outside the range, and the noise is amplified into “randomly wiping out the learning signal of a subset of tokens”. Chapter 6 of the RLHF Book gives two concrete descriptions of the failure as it plays out: a single token with a large ratio can dominate the entire update; and tokens within the same response are clipped independently of one another, shredding the learning signal.\nThe GSPO paper distills the crux into a single sentence: the unit of the optimization objective should match the unit of the reward. In RLVR, the reward is assigned to the entire sequence (1 point for a correct answer, 0 for a wrong one), whereas GRPO’s correction and clipping both happen at the token level. With the units mismatched, the more “conscientious” the correction, the more precisely targeted the damage.\nTo be honest, I should add one point here: GRPO’s formulation is not without theoretical grounding. Treating each token as one action step and decomposing the IS ratio of the whole trajectory into per-step ratios is the standard derivation path in RL. The problem lies at the implementation level, where the per-token ratios are used one by one as independent correction weights and clipped one by one — an approximation that stops being harmless under large vocabularies, long trajectories, and single samples. This is the GSPO paper’s critical angle, not an axiom; readers are free to reach their own judgment.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"Why Importance Sampling \"Fails\"","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"gspo-moving-ratio-clipping-and-optimization-to-the-sequence-level","content":"The GSPO objective:\n\\mathcal{J}_{GSPO}(\\theta)=\\mathbb{E}_{x\\sim\\mathcal{D},\\{y_i\\}_{i=1}^G\\sim\\pi_{\\theta_{old}}}\\left[\\frac{1}{G}\\sum_{i=1}^G \\min\\left(s_i(\\theta)\\hat{A}_i,\\ \\mathrm{clip}(s_i(\\theta),1-\\varepsilon,1+\\varepsilon)\\hat{A}_i\\right)\\right]\nThe difference from GRPO is visible at a glance: the per-token sum \\frac{1}{|y_i|}\\sum_t outside the min-clip is gone, and each response as a whole occupies just one term in the objective. The advantage is still the group-normalized \\hat{A}_i (standardized with the mean and standard deviation of the rewards within the group), shared by the entire response.\nThe core is the definition of the sequence-level ratio s_i:\ns_i(\\theta)=\\left(\\frac{\\pi_\\theta(y_i|x)}{\\pi_{\\theta_{old}}(y_i|x)}\\right)^{\\frac{1}{|y_i|}}=\\exp\\left(\\frac{1}{|y_i|}\\sum_{t=1}^{|y_i|}\\log\\frac{\\pi_\\theta(y_{i,t}|x,y_{i,\u003ct})}{\\pi_{\\theta_{old}}(y_{i,t}|x,y_{i,\u003ct})}\\right)\nThis is the geometric mean of the whole-sequence likelihood ratio, equivalent to taking the arithmetic mean of the per-token log-ratios and then exponentiating. The length normalization 1/|y_i| cannot be dropped: without it, thousands of ratios get multiplied together, likelihood changes in just a few tokens can make the overall ratio swing violently, and responses of different lengths would need different clipping ranges — there would be no way to set \\varepsilon at all. After normalization, the numerical range of s_i is decoupled from sequence length, and a single \\varepsilon covers everything. (The hands-on-modern-rl textbook omits this 1/|y_i| from its formula when introducing GSPO in Section 16.4; go by Equation 7 in the paper — leave it out and the whole chain of argument breaks.)\nThe meaning of clipping changes accordingly. In GRPO, clipping zeroes out the gradient of an individual token — the signal is chopped up, but the sample remains; in GSPO, clipping removes an entire response from the gradient estimate — a sample is either used in full or not at all. This aligns, in granularity, with “the reward is given to the whole sequence”.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"GSPO: Moving Ratio, Clipping, and Optimization to the Sequence Level","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"the-gradient-view-unequal-vs-equal-weighting","content":"Expand the gradients of the two objectives (Equations 8–12 in the paper), and the fundamental difference comes down to one thing: what multiplies each token’s \\nabla_\\theta\\log\\pi_\\theta.\nIn GRPO’s gradient, token t’s \\nabla\\log\\pi is multiplied by its own ratio w_{i,t}. After the min-clip, these per-token weights lie in (0, 1+\\varepsilon] (when \\hat{A}_i\u003e0) or [1-\\varepsilon, +\\infty) (when \\hat{A}_i\u003c0) — within the same response, the gradient weights that different tokens receive can differ by orders of magnitude, and as they accumulate over training steps, the consequences are hard to predict.\nIn GSPO’s gradient, all tokens of the same response share the same scalar weight s_i:\n\\nabla_\\theta\\mathcal{J}_{GSPO} = \\mathbb{E}\\left[\\frac{1}{G}\\sum_{i=1}^G s_i(\\theta)\\hat{A}_i \\cdot \\frac{1}{|y_i|}\\sum_{t=1}^{|y_i|} \\nabla_\\theta\\log\\pi_\\theta(y_{i,t}|x,y_{i,\u003ct})\\right]\nThe tokens’ contributions to the update differ only in the magnitude of their own \\nabla\\log\\pi; there is no longer any per-token random weight modulating them. The paper describes this as eliminating the destabilizing factor in GRPO.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"The Gradient View: Unequal vs. Equal Weighting","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"evidence-three-experimental-facts","content":"The paper’s experimental setup is itself worth noting: a cold-start model (obtained by SFT from Qwen3-30B-A3B-Base), with each rollout batch split into 4 mini-batches for gradient updates — a standard off-policy setting, and precisely the setting in which, as the previous post on Clip-Higher put it, “clipping actually starts to do its job” (only then does the ratio deviate from 1). The GRPO baseline’s clipping range of 0.2/0.27 was carefully tuned; GSPO uses 3e-4 / 4e-4.\nFact one: the three-orders-of-magnitude gap in ε is not a typo; it is an inevitable consequence of averaging. A single token-level ratio can easily shoot up to 1.2; the sequence-level ratio is the exp of the average of the per-token log-ratios, and averaging cancels out positive and negative fluctuations, so s_i naturally stays close to 1, fluctuating only slightly. This can be worked out in detail (what follows is my own estimate; the paper does not report these statistics): suppose that after one mini-batch update, each token’s log-ratio drift has, on average, a standard deviation of \\sigma (say, 5%), the response length is L, and the drifts of different tokens are approximately independent; then the standard deviation of the averaged drift shrinks to \\sigma/\\sqrt{L} — for L=2000, a reduction to about 1/45. Give such a nearly motionless ratio a clipping band of \\pm 0.2 and it will never touch the edges; clipping might as well not exist. So GSPO has to tighten the band to a few parts in ten thousand: 0.2 / 3\\times10^{-4} \\approx 667 — that is where the three-orders-of-magnitude narrowing comes from. Put another way: a tolerance band of 3e-4 corresponds to allowing an average log drift of only about 0.03% per token. I should also be upfront about how sound this assumption is: in real training, per-token drifts are not independent (a single update raises or lowers the probabilities of similar contexts en masse), so the actual reduction factor falls short of \\sqrt{L} — but this only makes the conclusion of Fact two below stronger.\nFact two: a hundred times more tokens get clipped, yet efficiency is higher. The band is a thousand times narrower, yet a hundred times more tokens are clipped — this sounds contradictory, but once you multiply two factors together, it isn’t (the estimates below are, as above, my own). First factor: relative to their respective fluctuation scales, the band is actually narrower. The fluctuation shrinks by only about 45×, while the band shrinks by 667×; using each ratio’s own fluctuation as the yardstick, GSPO’s band is about 15× narrower than GRPO’s (\\sqrt{L}/667), so s_i goes out of bounds far more easily than w_t does. Plugging in concrete numbers (\\sigma=5\\%, approximately Gaussian drift): GRPO’s 0.2 corresponds to 4\\sigma, so the probability of a single token going out of bounds is a few in ten thousand; GSPO’s 3e-4 corresponds to only about 0.27\\sigma (0.0003/(0.05/45)), so each response has a better-than-even chance of being thrown out in its entirety. Second factor: going out of bounds is charged to the whole response. In GRPO, a token that goes out of bounds discards only itself; in GSPO, a single out-of-bounds event removes the entire span of several thousand tokens. Multiply the two factors, and the total number of clipped tokens comes to roughly a hundred times GRPO’s — the gap in clipping fractions measured in Figure 2 of the paper is indeed about two orders of magnitude, so the magnitudes line up. The truly counterintuitive part is the second half of the claim: fewer tokens are used for training, yet training efficiency and benchmark performance are better (winning on all three curves: AIME'24 / LiveCodeBench / CodeForces). The paper’s interpretation: GRPO’s token-level gradient estimates are noisy to begin with, and the portion that happens to survive is no more useful than the portion that gets clipped; GSPO’s sequence-level filtering provides a more reliable learning signal — fewer but clean beats more but dirty.\nFact three: MoE no longer needs Routing Replay. The sequence likelihood \\pi_\\theta(y_i|x) is insensitive to routing jitter — when 10% of the experts change, individual token probabilities may shift noticeably, but the likelihood of the whole sequence (a reflection of the model’s language-modeling capability) does not change drastically. GSPO cares only about the latter, so simply computing the ratio as usual is enough for it to converge; the patch’s memory and communication overhead is saved entirely, and the model is no longer bound to the old routes.\nThere is also an infrastructure dividend on the side (§5.4 of the paper): in practice, the training engine (Megatron) and the inference engines (SGLang, vLLM) differ in numerical precision, so GRPO has to recompute the old policy’s likelihoods with the training engine before it can safely use them; sequence-level likelihoods are far more tolerant of precision discrepancies, so GSPO can optimize directly with the likelihoods returned by the inference engine and skip the recomputation. For partial rollouts, multi-turn RL, and architectures that decouple training from inference, this is a tangible throughput dividend.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"Evidence: Three Experimental Facts","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"appendix-how-the-3e-44e-4-clipping-range-was-set","content":"At the level of the paper: it isn’t stated. The GSPO paper gives only the result (left bound 3e-4, right bound 4e-4), with no selection procedure and no ablation on the clipping range. Why this value rather than 1e-4 or 1e-3 is impossible for outsiders to know; a reasonable guess is that the authors found it by trial and error on their own setup (cold-start Qwen3-30B-A3B, each batch split into 4 mini-batches), but that is only a guess.\nAt the level of principle (what follows is my inference, not the paper’s content): the clipping range’s job is to let normal fluctuations through and stop abnormal ones, so it should be set at a small multiple of the typical fluctuation magnitude of s_i:\n\\varepsilon \\approx k \\cdot \\frac{\\sigma}{\\sqrt{L}}\nHere \\sigma is the magnitude of the per-token log-ratio drift after one update, L is the response length, and k is the multiple of the fluctuation you want to let through. Plugging in the illustrative numbers from earlier (\\sigma=5\\%, L=2000) gives \\sigma/\\sqrt{L} \\approx 0.0011, so the paper’s 3e-4 corresponds to k \\approx 0.27 — letting through only about a quarter of the typical fluctuation, a very strict filter, which squares with the measured “tens of percent of tokens clipped”.\nThis framework also shows that 3e-4 cannot be copied as-is; it is coupled to three things: the degree of off-policyness (the staler the data, the larger \\sigma, and \\varepsilon must be loosened accordingly, or else every sample gets clipped away); response length (the longer L, the smaller the fluctuation after averaging, so the same \\varepsilon becomes stricter — suitable values for 32k and 2k differ by about 4×); and the desired clipping strength (there is no public data on the optimal clipping fraction, so a reproduction has to run its own sweep). Change the configuration, and this number has to be set again.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"Appendix: How the 3e-4/4e-4 Clipping Range Was Set","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"my-three-reservations","content":"First, the “single-sample” critique applies equally to GSPO itself. s_i is also a ratio computed from a single sample y_i, likewise without any N \\gg 1 averaging. So GSPO’s improvement should not be understood as “restoring the theoretical correctness of importance sampling” — it does no such thing; the paper’s IS argument is intuitive, not formal. What it really does is bring three things together: unit alignment (what gets clipped is an entire “bad sample” rather than random tokens), variance normalization (1/|y| squeezes the numerical range onto a common scale), and equal-weight gradients (eliminating the accumulation of unequal per-token weights). All three are engineering arguments — solid, but not theorems.\nSecond, the geometric mean is a diluter, and that cuts both ways. In a 10,000-token CoT, if the probability of one critical token drifts by 50%, its contribution to s_i is only e^{\\log 1.5 / 10000} \\approx 1.00004 — nearly invisible. Against precision noise between the training and inference engines, dilution is a feature (this is exactly where the infrastructure dividend comes from); for detecting “a critical token altered by a policy update”, dilution is a loophole. GSPO works properly precisely because it is paired with an extremely narrow clipping band on the order of three parts in ten thousand — making trust-region judgments based on average drift rather than single-point drift. Whether this combination goes blind to certain failure modes on ultra-long CoTs (for example, the policy deviating sharply on only a short stretch of reasoning, which gets let through after averaging) is something the paper does not answer.\nThird, sequence-level clipping gives up on making use of “partially correct” samples. If a response gets 90% of its reasoning right but miscalculates in the final step, GSPO either uses it whole or throws it out whole. The paper’s way out is the GSPO-token variant: using stop-gradient, it combines the sequence-level weight s_i (taking only its value, without backpropagating gradients through it) with the per-token term \\pi_\\theta / \\mathrm{sg}[\\pi_\\theta]; the resulting objective, clipping condition, and theoretical gradient are fully equivalent to GSPO’s, but advantages can be customized per token — an opening left for multi-turn RL and fine-grained rewards. At the other end of this line is VAPO: bring the value model back outright to do token-level credit assignment. Sequence-level vs. token-level is not a dispute over right and wrong, but two industrial answers to the old question of “credit-assignment granularity”.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"My Three Reservations","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"how-it-relates-to-clip-higher-and-a-detail-no-one-has-pointed-out","content":"The previous post covered DAPO’s Clip-Higher: the pathology is the same — a structural flaw of token-level clipping under off-policy training. DAPO’s remedy is to adjust the bounds (raising the upper bound from 0.2 to 0.28 to cut exploratory tokens some slack); GSPO’s remedy is to change the unit (the whole response shares a single ratio). The two are not mutually exclusive — Qwen’s practice is precisely GSPO stacked with a Dr.GRPO-style bias correction that removes the normalization.\nThere is one more detail worth pointing out: the GSPO paper’s clipping range of 3e-4 / 4e-4 is itself asymmetric — the right bound is wider than the left, leaving 33% more room in the reinforcing direction (the ratio breaking upward when \\hat{A}\u003e0). Isn’t that just a sequence-level version of Clip-Higher? The paper does not discuss where this asymmetry comes from, but its direction is consistent with DAPO’s logic: let good samples go a little further. Two teams independently prescribed remedies pointing in the same direction for the same pathology — that says more than any single paper’s ablation could.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"How It Relates to Clip-Higher, and a Detail No One Has Pointed Out","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"open-questions","content":"The method for choosing \\varepsilon has itself not been disclosed: 3e-4/4e-4 comes with results but no ablation, and \\varepsilon \\approx k \\cdot \\sigma/\\sqrt{L} is only this post’s inferred framework; the optimal range of k, and the effect of combining it with clip-higher-style asymmetric bounds, all call for public reproduction; Whether the dilution effect of the geometric mean systematically lets “abrupt local shifts” slip through on ultra-long CoTs (beyond 32k tokens) needs to be verified with purpose-built experiments; The actual effectiveness of GSPO-token paired with turn-level rewards in multi-turn agentic RL is, for now, backed only by Qwen’s internal data. Update log (2026-08-29): Expanded based on feedback from three rounds of reader Q\u0026A — the “Evidence” section now includes the full derivation of the clipping-band arithmetic (\\sigma/\\sqrt{L} variance reduction, the 667× narrowing, the band being ~15× narrower in relative terms, and the out-of-bounds probability calculation); a new section, “Appendix: How the 3e-4/4e-4 Clipping Range Was Set”, was added (the paper does not disclose its selection method; an inferred framework and three couplings); the first open question was rewritten accordingly. All derivations are labeled estimates, not content from the paper.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"Open Questions","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"references","content":"GSPO paper: arXiv:2507.18071v2; the equation numbers and experimental data in this post all come from its v2 RLHF Book, Chapter 6 (importance sampling background; GSPO/CISPO subsections): rlhfbook.com; book/chapters/06-policy-gradients.md hands-on-modern-rl, Section 16.4 (comparison of the GRPO family of improvements; note that its GSPO formula omits the length normalization): docs/chapter18_grpo/grpo-family.md Previous post: Entropy Collapse: What DAPO’s Clip-Higher Fixes","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/","section":"References","summary":"Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.","tags":["rl","post-training","grpo","gspo","moe"],"title":"The Ratio's Wrong Unit: GSPO Goes Sequence-Level"},{"anchor":"","content":"When naive GRPO is used to reproduce R1, entropy collapses rapidly and samples within a group converge; DAPO’s Clip-Higher merely raises the upper clipping bound from 0.2 to 0.28. Behind that one number lies a structural asymmetry of PPO-Clip on the probability axis. This post unpacks the failure chain, the mechanism, the evidence, and the limits.\nThis post has a companion interactive report (in Chinese), with drill-down breakdowns of the formulas and charts of the ablation data.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"Entropy Collapse: What DAPO's Clip-Higher Fixes","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"a-scene-from-a-training-run","content":"In early 2025, many teams were doing the same thing: reproducing DeepSeek-R1’s training with GRPO. The DAPO paper (arXiv:2503.14476) reports the authors’ own starting point—running naive GRPO on the Qwen2.5-32B base model scored only 30 on AIME 2024, while DeepSeek’s R1-Zero-Qwen-32B scored 47.\nThe gap did not open up gradually; it was a very specific way of dying: during training, the actor model’s entropy drops rapidly, the responses sampled as a group for the same problem grow more and more alike, and in the end they are nearly identical. The paper calls this entropy collapse. Sampling loses its diversity, rewards within the group converge, GRPO’s advantages all approach zero, and the gradient signal extinguishes itself. Training is still running, but it can no longer learn anything.\nThe first remedy DAPO prescribes is called Clip-Higher: raise PPO-Clip’s upper clipping bound ε from the default 0.2 to 0.28 and leave the lower bound at 0.2. The change is a single number. Why can this one number stop the collapse? The answer lies in a structural property of PPO-Clip that is rarely spelled out.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"A Scene from a Training Run","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"first-pinning-down-entropy","content":"This whole post revolves around a single word—“entropy”—so let me pin down its definition first. The entropy here is exactly the entropy of information theory, and the object it applies to is the probability distribution over the entire vocabulary that the model outputs each time it generates a token:\nH = -\\sum_{x \\in \\text{vocab}} p(x) \\log p(x)\nThe flatter the distribution (many tokens with similar probabilities), the higher the entropy, and the more the model “can’t make up its mind”; the sharper the distribution (one token takes 0.99 and the rest split 0.01), the lower the entropy, and the more “certain” the model is. A numerical example: suppose the vocabulary has only 3 tokens; the distribution [0.9, 0.05, 0.05] has an entropy of about 0.39 nat, while the uniform distribution [1/3, 1/3, 1/3] has about 1.10 nat. Sample 10 times from the former and you get the same token 9 times; samples from the latter spread out evenly—entropy is the mathematical measure of “sampling diversity”. The “actor model entropy” that the DAPO paper monitors is the average of these per-step distribution entropies over rollout generation.\nWhy it is the lifeline of RL training: all of GRPO’s training signal comes from the within-group reward variance, and that variance comes from sampling diversity. With high entropy, 16 responses sampled for the same problem all differ, some right and some wrong, and the advantages are nonzero; when entropy collapses, the 16 converge, the rewards are all identical, and the advantages are all zero.\nTwo caveats, so as not to over-read this: first, higher entropy is not always better—excessively high entropy corresponds to runaway modes such as gibberish and repeated words; the paper only says entropy should be kept within an “appropriate range”, without saying how to determine that range (one of the open questions at the end of this post). Second, don’t conflate entropy with sampling temperature: entropy is the intrinsic shape of the distribution, while temperature is how you draw from it; it takes parameters like temperature 1.0 at rollout time to cash entropy out as diversity. Clip-Higher affects the former—it keeps training itself from squeezing the distribution ever sharper.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"First, Pinning Down \"Entropy\"","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"first-working-out-the-ppo-clip-math","content":"The PPO/GRPO objective, viewed token by token, is:\nJ(\\theta) = \\mathbb{E}\\left[\\min\\left(r_t(\\theta)\\,\\hat{A}_t,\\ \\text{clip}(r_t(\\theta),\\ 1-\\varepsilon,\\ 1+\\varepsilon)\\,\\hat{A}_t\\right)\\right]\nThe symbols, one by one:\nr_t(\\theta) = \\pi_\\theta(o_t \\mid q, o_{\u003ct}) / \\pi_{\\theta_{old}}(o_t \\mid q, o_{\u003ct}), the importance sampling ratio: the current policy’s probability for the t-th token divided by the old policy’s probability at sampling time. r=1 means the policy has not changed. \\hat{A}_t, the advantage: how much better this token is than average. In GRPO it is computed by within-group normalization: \\hat{A}_{i,t} = (R_i - \\text{mean}(\\{R\\})) / \\text{std}(\\{R\\}). \\varepsilon, the clipping range: how far the ratio is allowed to deviate from 1; the default in the vast majority of implementations is 0.2—a default that comes from the Atari and robotic-control experiments in the original 2017 PPO paper. On where the advantage comes from, PPO and GRPO take two different paths; don’t mix them up when reading the literature:\nPPO: relies on a learned value function (the critic) and estimates the advantage token by token with GAE: \\hat{A}_t = \\sum_{l}(\\gamma\\lambda)^l \\delta_{t+l}, where \\delta_t = r_t + \\gamma V(s_{t+1}) - V(s_t). If the critic is accurate, the advantage has low variance; if the critic is off, the whole training run goes off with it. GRPO: no critic; it samples G responses for the same problem and normalizes with the within-group rewards. In essence it uses the group mean as the baseline—this is REINFORCE’s variance-reduction trick, except that the baseline doesn’t have to be learned; it is computed directly from the samples in the same group. The GRPO route has three direct consequences, all of which come up later. First, identical rewards across a group yield all zeros: when all G responses are right or all are wrong, the mean equals each response’s own reward, every advantage is 0, and the problem contributes no gradient at all. Second, dividing by std carries an implicit weighting: groups with a small std (say, only 1 of 16 correct) get their weight amplified by the normalization, and the difficulty bias that Dr.GRPO criticizes comes from this term. Third, the advantage is computed per whole response and shared by every token in it: in a wrong response, the first 80% of the reasoning steps may well be correct, yet they receive the same negative advantage as the final step where the calculation went wrong—GRPO by itself has no token-level credit assignment.\nHow the \\min behaves depends on the sign of \\hat{A}—this is the key to understanding everything:\n\\hat{A} \u003e 0 (good token, to be reinforced): when r \u003e 1+\\varepsilon, the clipped branch is selected, the objective becomes a constant, and the gradient is zero. In other words, each update step can raise a token’s probability to at most 1+\\varepsilon times its value at sampling time; beyond that, no gradient is given. \\hat{A} \u003c 0 (bad token, to be suppressed): symmetrically, each step can push the probability down at most to 1-\\varepsilon times its value at sampling time. There is an easily misread point here that is worth unpacking: for an advantage of either sign, the clip is one-sided. Take the case \\hat{A} \u003e 0 and r \u003c 1-\\varepsilon: the \\min still selects r \\cdot \\hat{A} (because it is the smaller one). Although the objective’s value is small, the gradient is alive—dJ/d\\pi \u003e 0, and the optimizer still pulls this good token’s probability back up with the full slope \\hat{A}. A small value does not mean suppression: the value is the outcome; the gradient is the action. What the \\min does is withdraw an incentive, never apply a reverse incentive. So the complete structure is: for good tokens, only the incentive to rise is capped (the plateau on the right); for bad tokens, only the incentive to suppress is capped (the plateau on the left); correction in the opposite direction always goes through with the full gradient.\nThe core mechanism chart in the interactive report is drawn on exactly this logic. How to read it: the horizontal axis is the ratio r, the vertical axis is this token’s contribution to the objective, J(r), and \\hat{A} is normalized to 1. For r \u003c 1+\\varepsilon, J(r)=r is a rising straight line (the gradient is alive); once r \u003e 1+\\varepsilon, it becomes a horizontal plateau (the gradient stops there).\nChapter 6 of the RLHF Book also makes a point of noting that the clip is not always at work. Only when the ratio actually goes out of bounds and the \\min happens to select the clipped branch is the gradient cut off. The clip fraction is the metric that monitors exactly this. Keep this in mind; it will come in handy later.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"First, Working Out the PPO-Clip Math","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"symmetric-clipping-asymmetric-effects","content":"Now for some arithmetic. Let \\varepsilon = 0.2 and \\hat{A} \u003e 0, and consider two tokens:\nA token with \\pi_{\\theta_{old}} = 0.9: the upper bound on its probability is 0.9 \\times 1.2 = 1.08. A probability cannot exceed 1, so it is effectively unconstrained—in one round it can be pushed from 0.9 to above 0.99. A token with \\pi_{\\theta_{old}} = 0.01: its upper bound is 0.01 \\times 1.2 = 0.012. An absolute increment of 0.002—it barely moves. First, let me pin down “one round” precisely: it means one “sample → update” cycle. After sampling, \\pi_{\\theta_{old}} is frozen as a snapshot, and then multiple gradient updates are run on this batch of data (16 for DAPO, paper §4.1); during this time \\pi_\\theta gradually drifts away from the snapshot and r starts drifting from 1. Once r crosses 1+\\varepsilon, that token’s gradient is cut off, and the remaining updates in this round no longer push it. So the precise meaning of “an upper bound of 0.012” is the point where the incentive vanishes, not a hard clamp—with a large step, the probability may overshoot it in a single step; it’s just that once past it, the incentive drops to zero: nothing pushes it further, and nothing pulls it back.\nAnd the budget refreshes every round: the next round resamples, \\pi_{\\theta_{old}} is updated, r resets to 1, and the token gets another \\times 1.2 budget. So a low-probability token grows along a multiplicative path: 0.01 \\to 0.012 \\to 0.0144 \\to \\cdots—exponential growth in ratio terms, yet in absolute terms it climbs very slowly through the first stretch; a high-probability token hits 1.0 and saturates within a round or two. The suppression direction is symmetric but harsher: with \\hat{A} \u003c 0, each round can cut the probability down to as little as \\times 0.8, leaving 0.33 after five rounds, so low-probability tokens are quickly beaten down to near 0—multiplicative decay on the way down is much faster than multiplicative growth on the way up.\nThis is exactly where the problem lies: clipping is measured on the “ratio” scale, but learning takes effect on the “probability” axis. With the same 20% tolerance on the ratio, a high-probability token gets close to 0.2 of room in absolute probability, while a low-probability token gets 0.002. The ranges of movement of the two kinds of token differ by nearly a hundredfold.\nAnd “exploration” in RL is carried precisely by low-probability tokens—tokens the current policy is not inclined to pick but that may lead to new solutions. DAPO’s measurements bear this out: the tokens actually clipped by the upper bound during training have an average probability below 0.2 (Figure 3a of the paper). In other words, the tokens the clipping mechanism keeps pinning down are precisely the exploration tokens; exploitation tokens (tokens the model is already very sure about) are almost never constrained.\nLong-CoT settings amplify this bias: a reasoning chain runs to thousands of tokens, an exploration token at a critical turning point can move only 0.2% per round, and before it has a chance to grow, the rest of the distribution has already sharpened further.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"Symmetric Clipping, Asymmetric Effects","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"the-closed-loop-of-collapse","content":"Put this micro-level bias into GRPO’s training dynamics and you get a complete failure chain:\nExploration tokens’ probabilities can’t grow, high-probability tokens keep climbing, the distribution sharpens, and entropy drops; A sharper distribution means the responses sampled as a group (say, 16) for the same problem become more and more alike; GRPO’s advantage is computed by within-group normalization; as within-group rewards converge, \\text{std} \\to 0 and all advantages approach zero; The gradient signal vanishes; the model loses its direction of improvement and can only keep reinforcing patterns it is already sure of—back to step 1. In the RLVR setting this loop is especially lethal, because its entire training signal is the within-group reward variance. The reward is a rule-based 0/1 (+1 for a correct answer, -1 for a wrong one), with no reward model to provide a continuous signal. Once diversity dries up, the signal source is cut off. Entropy collapse is not “a monitoring metric that looks bad”; it is training burning its own fuel.\nThis also explains the logic behind the ordering of DAPO’s four techniques: Clip-Higher prevents the collapse, while Dynamic Sampling (filtering out prompts whose samples are all correct or all wrong) stanches the bleeding once the collapse has already produced zero-gradient groups. One treats the root cause, the other the symptoms; the paper puts Clip-Higher in the first section.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"The Closed Loop of Collapse","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"clip-higher-raising-only-the-upper-bound-is-deliberate","content":"What Clip-Higher does is split the symmetric clip into two hyperparameters:\n\\text{clip}\\left(r_t(\\theta),\\ 1-\\varepsilon_{low},\\ 1+\\varepsilon_{high}\\right),\\quad \\varepsilon_{low}=0.2,\\ \\varepsilon_{high}=0.28\nThe upper bound goes from 1.2 to 1.28, so a low-probability exploration token’s room to rise in a single round goes from +20% to +28%; compounded by the multiplicative accumulation of the budget across rounds, the effect is considerable.\nThe question worth asking is: why not touch the lower bound? The paper is blunt about it—if \\varepsilon_{low} were loosened (say, allowing probabilities to be pushed down to 0.6×), the gradient from negative advantages would, by the same ratio logic, push low-probability tokens straight down to near 0; those tokens would effectively be deleted from the sampling space, and the collapse would only come faster. We did the multiplicative-decay math earlier: the downward direction is already faster than the upward one, so clipping in the suppression direction should, if anything, be tightened, not loosened.\nSo Clip-Higher is not some vague move like “opening the door to exploration a little wider”; it is a precise piece of asymmetric surgery: loosen the restraints on the way up, stay on guard on the way down.\nIn terms of results, Figure 2 of the paper shows that after Clip-Higher is applied, the entropy curve clearly recovers and AIME accuracy rises in step. The ablation chain (Qwen2.5-32B, AIME 2024 avg@32):\nConfiguration AIME score Naive GRPO 30 + Overlong Filtering 36 + Clip-Higher 38 + Soft Overlong Punishment 41 + Token-level Loss 42 + Dynamic Sampling (full DAPO) 50 To be honest: Clip-Higher on its own contributes only +2 points, not the largest gain in the chain (Overlong Filtering gives +6, and adding Dynamic Sampling at the end gives +8). But its role isn’t to add points—what it fixes is the stall point of “whether training can keep scaling”. Once entropy collapses, the three techniques that follow have nothing to work with.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"Clip-Higher: Raising Only the Upper Bound Is Deliberate","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"two-premises-rarely-spelled-out","content":"Premise one: the clip activates only when training is off-policy. Chapter 6 of the RLHF Book points out that in RL practice for language models, many algorithms take only one gradient step per batch—in that case the ratio is identically 1, the clip never triggers, and what actually acts as the regularizer is the KL penalty. DAPO, by contrast, takes 16 gradient steps per rollout (mini-batch 512, paper §4.1): the policy is updated multiple times within a batch, and only then does the ratio deviate from 1 and the clip actually start working. So the value of the Clip-Higher knob is coupled to how off-policy the training is: if your training is strictly single-step on-policy, this knob simply does not exist. Conversely, asynchronous RL (where rollout and training overlap and samples are inherently off-policy) is the soil in which it really comes into play.\nPremise two: DAPO also drops the KL term. First, let me be clear about the KL penalty itself. The KL divergence measures how far the current policy \\pi_\\theta is from the reference policy \\pi_{ref} (usually the frozen SFT/base model):\nD_{KL}(\\pi_\\theta \\| \\pi_{ref}) = \\mathbb{E}_{x \\sim \\pi_\\theta}\\left[\\log \\pi_\\theta(x) - \\log \\pi_{ref}(x)\\right]\nClassic RLHF puts it into the objective as a penalty term (-\\beta D_{KL}), for two reasons: first, to prevent reward hacking—the reward model is a leaky proxy, and a policy that strays far enough can find outputs the RM has never seen and blindly scores high; the KL tethers the search to the region of the distribution the RM has seen; second, to preserve the initial model’s language abilities and keep formatting from breaking down. The engineering cost is that a frozen reference model has to stay resident in GPU memory, and every token needs a forward pass through both models.\nDAPO removes it entirely (\\beta=0, so the reference model doesn’t need to be loaded), with the reasoning given in §2.3 of the paper: the KL penalty was originally meant to “align, but don’t stray too far from the initial model”, whereas the goal of long-CoT training is precisely to depart substantially from the initial model—the base model cannot write long chains of thought or reflect on itself to begin with; these behaviors depend entirely on RL reshaping the distribution, and staying tethered to the anchor amounts to not letting the ship sail. There is also an unstated enabling condition: RLVR rewards are rule-based judgments, so there is no RM to hack, and the anti-hacking half of the KL’s motivation simply disappears.\nPutting the two premises together reveals an implicit gear shift in DAPO: in the classic setup, drift is held in check by the KL—an absolute anchor that forbids straying too far from the initial model; once DAPO drops the KL, only the clip remains—a relative anchor that forbids each step from straying too far from the previous one but allows the policy to keep moving farther away. The trust-region role has passed from one anchor to the other. Note also that dropping the KL and raising the upper bound point in the same direction—both loosen the restraints on exploration: the KL penalty pulls the distribution toward the reference model token by token, and only with it gone does the policy dare to venture into new regions. The paper does not say outright that the two were designed to work together—this is my inference; but judged by the goal of “preventing collapse”, they are indeed fighting in the same trench. Of course, removing the anchor is not free: when training spins out of control, nothing pulls the model back, which is why DAPO relies so heavily on its monitoring dashboard (entropy, length, mean probability), and why it needs substitute stabilizers such as the clip and dynamic sampling.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"Two Premises Rarely Spelled Out","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"cross-checking-the-implementations","content":"Beyond the mechanism on paper, I checked three implementations, and the details largely line up:\nrlhf-book’s DAPOLoss in code/policy_gradients/loss.py: decoupled clip (clip_eps_lo=0.2 / clip_eps_hi=0.28, see configs/dapo.yaml), token-level loss normalization (the loss is summed and then divided by the total number of valid tokens across the whole batch, rather than GRPO’s original per-sequence mean followed by a mean over sequences), and no KL. The teaching implementation is consistent with the paper. verl’s production implementation (verl/trainer/ppo/core_algos.py): clip_ratio_low / clip_ratio_high are independent config options that fall back to the symmetric clip_ratio when unset—that is, DAPO’s asymmetric clipping is already a first-class citizen, but the default behavior is still classic PPO. The file also contains dual-clip’s clip_ratio_c=3.0, which guards against ratio blow-up on negative-advantage samples—a problem in a different direction from the one Clip-Higher solves. Industry corroboration: MiniMax’s CISPO likewise allows asymmetric clipping bounds; VAPO adopts clip-higher directly. This insight is no longer unique to DAPO; it has become an engineering consensus in long-CoT RL. Let me also mention a fact-checking case: the Chinese-language course hands-on-modern-rl, in its explanation of Clip-Higher, says “a lower bound of 0.8 means a low-probability action can be pushed down to 0.008 at most, almost completely suppressed”, conflating the suppressive effect of the lower bound with the room to rise under the upper bound, which can easily lead readers to think it is the lower bound that was changed. The paper’s actual mechanism is exactly the opposite: what changes is the upper bound; the lower bound stays at 0.2 precisely because loosening it further would crush low-probability tokens. The course states that it was generated with AI assistance and has not yet been fully reviewed; when reading secondhand material, always check back against the original—this very check turned up one such error.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"Cross-Checking the Implementations","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"addendum-rlvr-and-grpo-are-not-on-the-same-layer","content":"When reading this post, it is easy to say RLVR and GRPO in one breath, but they are actually two orthogonal axes:\nQuestion it answers Which layer Other options on the same layer RLVR Where does the reward come from? Reward source / training paradigm Human preferences (RLHF’s starting point), learned reward models, verifiable rule-based rewards GRPO Given the reward, how do you update the model? Policy optimization algorithm REINFORCE, PPO, DAPO, GSPO The two can be combined freely: RLVR + PPO works perfectly well (VAPO is exactly the value-based route running on verifiable rewards); RM + GRPO holds up too (when GRPO came out of DeepSeekMath it was not tied to rule-based rewards, and within-group normalization works for any scalar reward). RLVR + GRPO became the mainstream recipe because the two bring out the best in each other: RLVR’s rule-based rewards are cheap, objective, and robust to reward hacking, but they are a sparse 0/1 signal; GRPO’s within-group normalization happens to suit 0/1 well—sample a group for the same problem and the ratio of right to wrong naturally forms a baseline, with no need to train a separate critic to estimate “how many points this line of reasoning is worth”. Conversely, PPO’s critic is hard to train accurately on sparse 0/1 rewards over sequences thousands of tokens long.\nPlaced back into this post’s failure chain, the division of labor between the two is clear as well: RLVR determines how fragile the signal source is (0/1 rewards come in only three patterns—all correct, all wrong, or a mix—and the first two give zero gradient outright), while GRPO determines what the signal depends on (within-group reward variance, i.e., sampling diversity). All four of DAPO’s techniques are patches at GRPO’s algorithm layer and leave RLVR’s reward source untouched; improvements in the other direction—building more verifiable tasks for RLVR, refining rewards down to the process level (PRM)—are surgery at the reward-paradigm layer, a path orthogonal to the algorithm-layer patches.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"Addendum: RLVR and GRPO Are Not on the Same Layer","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"my-assessment-and-open-questions","content":"Assessment: At its core, Clip-Higher turns the “pace of exploration” from an implicit default left over from history (ε=0.2, from the continuous-control settings of 2017) into an explicit knob geared to long-CoT reasoning. The 0.28 is not theoretically derived; it is an empirical value, but the diagnostic framework behind it—the asymmetry of ratio clipping on the probability axis—has general explanatory power. From now on, whenever you see “training was going fine, then entropy suddenly collapsed”, first checking the probability distribution of the tokens held down by the clip should become standard practice.\nOpen questions, in order of how much I care about them:\nEntropy has no target range. The paper’s empirical conclusion is that “a slow rise in entropy is beneficial”, but it also acknowledges that excessively high entropy corresponds to gibberish and repetition. So should ε_high be adjusted adaptively along with the entropy curve rather than fixed at 0.28? I haven’t seen anyone study this systematically. Order effects in the ablation. Table 1 is a cumulative chain: Clip-Higher is tested after Overlong Filtering, so its +2 points were measured on a base that already includes earlier changes. What its clean, standalone contribution is remains unknown. Compatibility with sequence-level clipping. GSPO lifts the importance sampling ratio to the sequence level (to fix the numerical instability of token-level ratios in MoE models); does the logic of “raising the token-level upper bound” still hold then? The GSPO post unpacks exactly this question. Robustness under asynchronous training. In asynchronous RL the ratio distribution drifts further and is more skewed; whether a fixed 0.28 is appropriate across different degrees of off-policyness is not covered by the paper.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"My Assessment and Open Questions","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"references","content":"DAPO paper: arXiv:2503.14476 (primary source for the mechanism, numerical examples, ablation data, and training details) RLHF Book Chapter 6 and its companion implementation: code/policy_gradients/loss.py, code/policy_gradients/configs/dapo.yaml (natolambert/rlhf-book) verl production implementation: verl/trainer/ppo/core_algos.py (clip_ratio_low/high, dual-clip) Bilibili video: Analysis of the GRPO loss/objective and its biases (DAPO, Dr. GRPO), BV1LgXbY5EFD (note: this episode mainly covers dynamic sampling and length bias, and Dr. GRPO’s standard-deviation issue; it spends very little time on Clip-Higher) hands-on-modern-rl docs/chapter18_grpo/deepseek-dapo.md (a Chinese-language explainer; the wording in its Clip-Higher section is muddled, as flagged in the main text) Notion “Common Language” notes page (the three rounds of industrial reworking of GRPO: DAPO / Dr.GRPO / GSPO) Update log: 2026-08-26, first version published; the same day, additions based on reader Q\u0026A—the definition and caveats of “entropy”, the two sources of the advantage in PPO and GRPO and their consequences, how to read the one-sided clipping of the min (value vs. gradient), the precise meaning of the “single-round budget” and its multiplicative accumulation, the layering of RLVR and GRPO, and a detailed look at the KL penalty and the absolute-anchor/relative-anchor gear shift.","date":"2026-08-26","permalink":"https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/","section":"References","summary":"Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.","tags":["rl","post-training","grpo","dapo","rlvr"],"title":"Entropy Collapse: What DAPO's Clip-Higher Fixes"},{"anchor":"jiaxing-ni","content":"19, working on the engineering side of LLM training — harness, evals, post-training, and every pipeline where the three meet.\nGood systems feel quiet on the surface. Underneath: rollout pipelines, reward traces, eval slices, and a few scripts watching the night shift. Most of my work is making those things tell the truth.","date":"","permalink":"https://njx-njx.github.io/en/about/","section":"Jiaxing Ni","summary":"About Jiaxing Ni","tags":[],"title":"About"},{"anchor":"what-i-work-on","content":"One through-line: turn real-world usage traces into signals that training can learn from.\nDig model failure patterns out of production sessions and user feedback, then forge them into benches and multi-turn data Build executable verifiers for agent tasks — layered scoring gates, mutation-test self-checks, judging noise that can be peeled out An eval owes more than a score; it should answer why it generalizes RL environments, rollout systems, SFT data synthesis — every gram of post-training signal should have a provenance","date":"","permalink":"https://njx-njx.github.io/en/about/","section":"What I Work On","summary":"About Jiaxing Ni","tags":[],"title":"About"},{"anchor":"traces","content":"Every stop so far circles the same problem: agent harness and swarm evals, distilling a model’s recurring failure modes out of a hundred-thousand-odd multi-turn sessions; a pretraining data foundation, plus an attempt to give a base model the “taste” to judge training recipes; the mechanics of evaluation itself — scores anchored to evidence, user satisfaction made interpretable through the geometry of a conversation; and further back, a system that gathered personal context scattered across macOS into something trustworthy, permissioned, and actionable.\nWhich banners these happened under, I’ll leave off the page — the work matters more than the names. Details are for offline conversation.","date":"","permalink":"https://njx-njx.github.io/en/about/","section":"Traces","summary":"About Jiaxing Ni","tags":[],"title":"About"},{"anchor":"coordinates","content":"Identity: researcher, builder, engineer. Base: Beijing / Shanghai. Taste: prototype quickly, measure honestly, keep only what survives contact with real edges.","date":"","permalink":"https://njx-njx.github.io/en/about/","section":"Coordinates","summary":"About Jiaxing Ni","tags":[],"title":"About"},{"anchor":"about-this-blog","content":"A field notebook on the mechanisms behind LLM training: what a knob actually does, what breaks when you turn it, and what the metric said.\nTopics orbit harness and rollout systems, model architecture, mid-training and post-training, evals and benchmarks, reasoning RL and agentic RL, RSI, and data synthesis. Longer posts often come with an interactive report edition, where formulas become sliders and curves become probes — see Reports.\nBuilt with Hugo and PaperMod, hosted on GitHub Pages, and open source on GitHub.","date":"","permalink":"https://njx-njx.github.io/en/about/","section":"About This Blog","summary":"About Jiaxing Ni","tags":[],"title":"About"},{"anchor":"faq","content":"What will you not write here? I try not to write generic AI commentary. If a post has no code, no experiment trace, no eval detail, or no honest uncertainty, it probably belongs somewhere else.\nCan I cite, translate, or share a post? Yes. Keep a link to the original post, and feel free to tell me where it traveled. I like seeing notes find another desk.\nAre you open to collaboration or conversation? Yes, especially around RL infrastructure (rollout and serving), post-training recipes, eval design, agentic systems, and strange little tools that make research easier to use.","date":"","permalink":"https://njx-njx.github.io/en/about/","section":"FAQ","summary":"About Jiaxing Ni","tags":[],"title":"About"},{"anchor":"find-me-online","content":"GitHub Hugging Face X Email: njxnjxai@gmail.com","date":"","permalink":"https://njx-njx.github.io/en/about/","section":"Find Me Online","summary":"About Jiaxing Ni","tags":[],"title":"About"}]