<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>agentic-rl on Njx&#39;Log</title>
    <link>https://njx-njx.github.io/en/tags/agentic-rl/</link>
    <description>Recent content in agentic-rl on Njx&#39;Log</description>
    <image>
      <title>Njx&#39;Log</title>
      <url>https://njx-njx.github.io/blog-cover.jpg</url>
      <link>https://njx-njx.github.io/blog-cover.jpg</link>
    </image>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate>Tue, 27 Oct 2026 15:14:17 +0800</lastBuildDate>
    <atom:link href="https://njx-njx.github.io/en/tags/agentic-rl/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>“The Model Trains Itself”: RSI and Three Loops</title>
      <link>https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/</link>
      <pubDate>Tue, 01 Sep 2026 12:30:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/</guid>
      <description>RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.</description>
      <content:encoded><![CDATA[<p>The term RSI (recursive self-improvement) became hot again in 2025-2026, but the things it refers to often get lumped together. Starting from a practitioner&rsquo;s dichotomy (change the harness but not the weights vs. change the weights directly), this post breaks “self-improvement” into three levels—the data loop, the reward loop, and the system loop—which differ in their preconditions, how far they have actually paid off, and their failure modes.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/rsi-self-improvement-loops/">interactive report</a>
 (in Chinese), with an interactive diagram of the three nested loops, a three-level exam matrix, and a scissors-gap diagram.</p>
</blockquote>
<h2 id="a-conflated-term">A Conflated Term</h2>
<p>&ldquo;RSI&rdquo; (Recursive Self-Improvement) became hot again in 2025–2026, but what people mean by it varies widely: some mean an iterative pipeline that filters rollouts into data and retrains on it, some mean a model rewriting its own training process, and some mean scaling up test-time compute. The term&rsquo;s degree of confusion is directly proportional to the allure of what it promises.</p>
<p>Let me start with a practitioner&rsquo;s framework. In a conversation, Ding Duo, head of post-training at Perplexity, split RSI and continual learning into two routes: <strong>one leaves the model parameters unchanged and optimizes the harness, the loop, and the solving process; the other updates the model weights directly</strong>. His judgment is that the former produces results faster, while the latter may bring deeper changes in capability; and the two routes share three hard problems—controlling cost, avoiding parameter contamination, and proving that what is continually learned is transferable capability. This post will keep coming back to these three problems.</p>
<p>Another common confusion is around the term test-time. Classic TTT (Test-Time Training, Sun et al. 2020) refers to taking gradient updates on the current input at inference time, whereas the &ldquo;test-time scaling&rdquo; popularized by s1 refers to lengthening the reasoning chain and adding budget, without touching the weights. When people in an LLM context talk about a &ldquo;new test-time paradigm&rdquo;, always ask which one they mean—the former actually changes the model; the latter just thinks more. What Ding Duo calls &ldquo;optimizing the harness/loop&rdquo; is in fact a third path: leave the gradients alone and change the scaffolding.</p>
<h2 id="level-one-the-data-loopmost-mature-and-least-like-rsi">Level One: The Data Loop—Most Mature, and Least Like RSI</h2>
<p>The first level is data circulating through the loop: model rollout → filter successful trajectories by verifiable outcomes → mix them into the next round of SFT/RL training → the model gets stronger → roll out again. LoopTool (<a href="https://arxiv.org/abs/2511.09148">arXiv:2511.09148</a>
) has exactly this shape (evaluation uncovers weaknesses → targeted data synthesis → retraining); a more complete industrial example is the WebSailor series (arXiv:2507.02592; the V2 version, 2509.13305, explicitly puts synthetic data + scalable RL in its title): use a strong model to generate browsing trajectories, train with outcome rewards, filter with self-critique, then automatically synthesize harder training tasks. DeepResearcher (2504.03160) reports a phenomenon worth remembering: in this kind of closed-loop training, behaviors such as pre-search planning, information layering, and cross-validation <strong>are not explicitly encouraged by any reward term; they are by-products of RL optimization</strong>.</p>
<p>This level is already industrialized, but strictly speaking it is not &ldquo;self&rdquo;-improvement—the choices in the loop are made by a filter designed by engineers, and the model is just a data source. Its ceiling is also clear: the filtering criteria themselves are hackable (inefficient trajectories that &ldquo;happened to succeed&rdquo; get kept), and the narrower the data distribution, the faster the loop turns, and the faster the model converges itself toward whatever the current verifier likes.</p>
<h2 id="level-two-the-reward-looprlsvr-aims-to-extend-the-self-verification-dividend-to-open-ended-tasks">Level Two: The Reward Loop—RLSVR Aims to Extend the Self-Verification Dividend to Open-Ended Tasks</h2>
<p>The second level is reward circulating through the loop. RLVR&rsquo;s success over the past two years rests on &ldquo;math and code have deterministic verifiers&rdquo;; open-ended tasks (writing, research, dialogue) have no verifier and can only rely on preference models or LLM judges, which brings a triple tax: evaluation bias, the judge-capability bottleneck, and inference cost. RLSVR (arXiv:2607.23802) borrows the concept of a &ldquo;pretext task&rdquo; from self-supervised learning: <strong>apply a task transformation to the open-ended task to turn it into a self-verifiable form</strong>, so that the reward once again comes from the task itself rather than from an external judge.</p>
<p>The conditions for this idea to hold are worth laying out: it requires that the task admit some fidelity-preserving transformation (getting the transformed task right ≈ doing the original task well), and that the verifier be deterministic in the transformed space. The &ldquo;task transformation induces self-verifiable rewards&rdquo; in the paper&rsquo;s title refers to exactly this inducing step. If it holds, open-ended tasks can also tap into the dividends of RLVR&rsquo;s industrialization; if it doesn&rsquo;t—if the transformation loses too much—the self-verifiable reward is optimizing a proxy metric that has come apart from the original task.</p>
<p>What this means for the loops: the reward loop is the most critical of the three levels for RSI. The data loop&rsquo;s filter and the system loop&rsquo;s evaluator both have to answer &ldquo;what counts as getting better&rdquo;, and self-verifiable reward is currently the only answer that does not depend on a continuous supply from humans.</p>
<h2 id="level-three-the-system-looprsi-proper-stuck-on-oversight">Level Three: The System Loop—RSI Proper, Stuck on Oversight</h2>
<p>Only the third level is RSI proper: the model (or what it produces) improves the things used to improve it—better assistants, better data, better evaluations—forming a recursion. Leike et al.&rsquo;s 2018 research agenda for reward modeling (arXiv:1811.07871) already has a section explicitly discussing recursive reward modeling: use AI to help humans evaluate AI outputs, so that evaluation capability grows along with whatever is being evaluated.</p>
<p>The prerequisite for this level is scalable oversight (2211.03540, cited in Chapter 30 of the hands-on-modern-rl textbook as a core direction in alignment): once model capability exceeds human evaluation capability, who verifies that an improvement is real? Every round of the loop makes selections, and once the selection criteria are hackable, <strong>the faster the loop turns, the faster it collapses</strong>—a small-scale version is already visible in the data loop (metric inflation), and the system loop amplifies it to the entire pipeline. This is also why &ldquo;proving that what is continually learned is transferable capability&rdquo; is on Ding Duo&rsquo;s list of hard problems: improvement inside the loop is often &ldquo;getting stronger on this loop&rsquo;s evaluations&rdquo;, which does not necessarily hold outside the loop.</p>
<h2 id="the-three-levels-share-one-exam">The Three Levels Share One Exam</h2>
<p>Putting the three levels together, each of Ding Duo&rsquo;s three hard problems maps onto concrete mechanisms:</p>
<ul>
<li><strong>Cost</strong>: the data loop&rsquo;s cost is rollout and filtering (every iteration samples in full; we did the rollout = prompt × G math earlier in the series); the reward loop&rsquo;s cost is constructing and maintaining the task transformations and verifiers; the system loop&rsquo;s cost is basically impossible to estimate for now—because no instance has yet run the full pipeline end to end.</li>
<li><strong>Parameter contamination</strong>: only level three is involved. Continual gradient updates let new knowledge interfere with old capabilities, and the contamination accumulates with the number of iterations—this is one reason the &ldquo;don&rsquo;t change the weights&rdquo; route is currently underrated: optimizing the harness and loop doesn&rsquo;t contaminate parameters, and the gains are immediate; engineering work like DeepSeek V3.2 (self-added verifiers, switching the sampler to mHC) is this route paying off.</li>
<li><strong>Transferability</strong>: none of the three levels has a good evaluation protocol. Evaluation inside a loop is contaminated by the loop itself (the training target and the evaluation target share the same source), and cross-loop evaluation is too expensive.</li>
</ul>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>the &ldquo;don&rsquo;t change the weights&rdquo; route is currently the main way RSI actually pays off, and theorists underrate it</strong>. Scaffold optimization (tools, context engineering, verifier internalization, sampling procedures) doesn&rsquo;t change parameters, doesn&rsquo;t contaminate weights, and can be rolled back, and at the current stage its return per unit of compute is higher than that of changing the weights directly. Expelling it from the ranks of &ldquo;self-improvement&rdquo; is a misallocation of resources caused by terminological purism.</p>
<p>Second, <strong>the three levels are bottlenecked in different places</strong>. The data loop is stuck on the hackability of its filter; the reward loop on the fidelity of the task transformation; the system loop on oversight (without a solution to scalable oversight, recursion can only be recursive self-deception). Discuss &ldquo;RSI progress&rdquo; without specifying the level, and the conclusions are bound to be vague.</p>
<p>Third, <strong>loop speed is a double-edged sword, and an amplifier of safety problems</strong>. Every paper on self-improvement talks about convergence speed, and few talk about runaway speed at the same time; yet the two share the same structure—the loop compounds &ldquo;small selection biases&rdquo;. For self-improving systems, what should be monitored is the <strong>scissors gap</strong> between in-loop and out-of-loop metrics, not any single metric.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>Whether there is a systematic way to construct task transformations for open-ended tasks (rather than hand-tuning them task by task); RLSVR only provides the paradigm;</li>
<li>Where the capability ceiling of the &ldquo;don&rsquo;t change the weights&rdquo; route lies—whether there are capabilities that can only be obtained by changing the weights; there is no clean evidence yet;</li>
<li>An evaluation protocol for cross-loop transferability: who evaluates, with what data, and how to keep the evaluation from being contaminated by any one of the loops.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Notes on a conversation with Ding Duo: 《在模型持续进步的时代，创业者应该站在哪里》§20 (the two routes for RSI and continual learning, and the three hard problems)</li>
<li>RLSVR: <a href="https://arxiv.org/abs/2607.23802">arXiv:2607.23802</a>
 (abstract and core claims)</li>
<li>WebSailor: <a href="https://arxiv.org/abs/2507.02592">arXiv:2507.02592</a>
, <a href="https://arxiv.org/abs/2509.13305">V2 2509.13305</a>
 (titles and positioning within the series); DeepResearcher <a href="https://arxiv.org/abs/2504.03160">arXiv:2504.03160</a>
</li>
<li>Recursive reward modeling: Leike et al. <a href="https://arxiv.org/abs/1811.07871">arXiv:1811.07871</a>
; Scalable Oversight <a href="https://arxiv.org/abs/2211.03540">arXiv:2211.03540</a>
 (hands-on-modern-rl textbook: <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter30_alignment_failures/scaling-and-defenses.md"><code>docs/chapter30_alignment_failures/scaling-and-defenses.md</code></a>
)</li>
<li>Classic Test-Time Training: Sun et al. <a href="https://arxiv.org/abs/1909.13231">arXiv:1909.13231</a>
; test-time scaling: s1 <a href="https://arxiv.org/abs/2501.19393">arXiv:2501.19393</a>
</li>
<li>Previous post: <a href="https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/">Who&rsquo;s to Blame When a 7-Turn Interaction Fails: Reward Credit Assignment for Multi-Turn Trajectories</a>
 (previous post: within-episode credit assignment; this post: cross-episode loops)</li>
</ul>
]]></content:encoded>
    </item>
    <item>
      <title>A 7-Turn Failure: Assigning Blame in GRPO/GSPO</title>
      <link>https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/</link>
      <pubDate>Mon, 31 Aug 2026 12:55:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/</guid>
      <description>Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token&amp;rsquo;s opening.</description>
      <content:encoded><![CDATA[<p>In single-turn RL the whole sequence shares one advantage, and nobody has ever lost sleep over it; in multi-turn agentic RL, the same trajectory can get the turn-1 search right, send the turn-2 query off course, and end with a wrong answer—so how to split a single trajectory reward becomes a real design decision. This post unpacks the formalism (action mask, trajectory probability), the credit-assignment spectrum (ORM/PRM/turn discounting/three middle-ground routes), and the opening GSPO-token leaves for multi-turn training.</p>
<blockquote>
<p>This post comes with a companion <a href="https://njx-njx.github.io/reports/turn-level-reward-credit-assignment/">interactive report</a>
 (in Chinese), featuring a toggleable ORM/PRM credit-assignment comparison, backward discounting with a γ slider, and a breakdown of the GSPO-token formula.</p>
</blockquote>
<h2 id="a-problem-that-doesnt-exist-in-single-turn-training">A Problem That Doesn&rsquo;t Exist in Single-Turn Training</h2>
<p>Consider a five-turn trajectory: turn 1&rsquo;s search hits the key evidence; turn 2 drifts off course while rewriting the query and brings back irrelevant content; turn 3 corrects course and gets halfway back; turn 4 makes a calculation error; turn 5 gives a wrong answer. The environment returns just one number: failure, reward = 0.</p>
<p>Single-turn RL doesn&rsquo;t have this problem—the entire response shares one advantage, and &ldquo;all right or all wrong&rdquo; is taken for granted. In the multi-turn setting this crude logic breaks down immediately: turn 1 clearly did the right thing, so why should it be penalized along with turn 4? On top of that there is a further engineering layer: the trajectory mixes two kinds of tokens, those the model generated itself (thinking, tool-call arguments, the final answer) and those returned by the environment (search results, code output), and the latter should never enter the gradient at all. So &ldquo;how does turn-level reward get into GRPO/GSPO&rdquo; is really three questions rolled into one: <strong>which tokens count as actions, how much advantage each turn gets, and which tokens that advantage gets multiplied onto</strong>. This post takes them apart in that order.</p>
<h2 id="first-pin-down-the-object-trajectory-probability-and-the-action-mask">First, Pin Down the Object: Trajectory Probability and the Action Mask</h2>
<p>In the formalization of multi-turn trajectories (the textbook adopts the POMDP formulation from the AppWorld paper, arXiv:2504.11536), the full state is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>t</mi></msub><mo>=</mo><mo stretchy="false">[</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>c</mi><mo separator="true">,</mo><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi></mrow></msub><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">z_t = [s_0, c, x_{1:t}]</annotation></semantics></math></span>: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">s_0</annotation></semantics></math></span> is the hidden initial environment state (database snapshot, REPL state), <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span> is the task context, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">x_{1:t}</annotation></semantics></math></span> is the entire token history so far—both what the model generated and what the environment returned. The model sees only the text part, which is why it is called partially observable.</p>
<p>The chain-rule factorization of the trajectory probability has one key detail:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>ρ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo>∣</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>c</mi><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="double-struck">I</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>x</mi><mo stretchy="false">)</mo><munder><mo>∏</mo><mrow><mi>t</mi><mo>∈</mo><mi>a</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow></munder><msub><mi>p</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>x</mi><mi>t</mi></msub><mo>∣</mo><mi>c</mi><mo separator="true">,</mo><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\rho_\theta(x \mid s_0, c) = \mathbb{I}(s_0, x) \prod_{t \in a(x)} p_\theta(x_t \mid c, x_{1:t-1})</annotation></semantics></math></span><p>The product runs <strong>only over <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a(x)</annotation></semantics></math></span>, the set of positions of model-generated tokens</strong>; the observation tokens returned by the environment are determined by the environment dynamics and are not part of the product. Translated into the loss, this is the action mask: a 0/1 vector the same length as the trajectory, with model tokens marked 1 and the prompt and tool returns marked 0. SearchR1&rsquo;s implementation makes this very explicit: the contents of <code>&lt;think&gt;</code> and <code>&lt;search&gt;</code> take part in training, while the <code>&lt;information&gt;</code> segments returned by the retriever are masked out entirely. Letting environment tokens into the gradient as well would amount to forcing the model to &ldquo;learn to predict which web pages the search engine returns&rdquo;, contaminating the policy gradient.</p>
<p>This sets the skeleton of this post&rsquo;s answer: turn-level reward enters training <strong>not by weighting environment tokens, but by constructing a separate advantage for each turn&rsquo;s model tokens</strong>.</p>
<h2 id="three-layers-of-signal-how-reward-travels-from-trajectory-to-weights">Three Layers of Signal: How Reward Travels from Trajectory to Weights</h2>
<p>The textbook splits the signal flow into three layers, and this layering is the map for understanding the whole spectrum of approaches:</p>
<ol>
<li><strong>Trajectory reward</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><msub><mi>τ</mi><mi>i</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">R(\tau_i)</annotation></semantics></math></span>: the raw scalar given by the environment—&ldquo;did this episode succeed or not&rdquo;;</li>
<li><strong>Step advantage</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>: breaks the final outcome down into per-turn scores, answering &ldquo;should turn <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span> be rewarded&rdquo;—this is the main battleground of credit assignment;</li>
<li><strong>Token gradient</strong>: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> is multiplied onto the log-probs of that turn&rsquo;s action tokens, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Δ</mi><mi>θ</mi><mo>=</mo><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><msub><mo>∑</mo><mrow><mi>k</mi><mo>∈</mo><msub><mi>a</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow></msub><mi mathvariant="normal">∇</mi><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>k</mi></msub><mo>∣</mo><mo>⋅</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Delta\theta = A_{i,t} \sum_{k \in a_{i,t}} \nabla \log \pi_\theta(y_k \mid \cdot)</annotation></semantics></math></span>.</li>
</ol>
<p>What &ldquo;putting turn-level reward into GRPO&rdquo; actually comes down to is layer 2: GRPO&rsquo;s within-group comparison naturally works at the trajectory level (ranking multiple trajectories for the same problem against each other), so to get down to the turn level you have to define <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> yourself and feed it in.</p>
<h2 id="why-the-naive-approach-fails-a-controlled-experiment">Why the Naive Approach Fails: A Controlled Experiment</h2>
<p>The textbook&rsquo;s Mini Agent Loop experiment gives the most convincing set of numbers. The task: look up Earth&rsquo;s radius → compute the equatorial circumference → verify → answer. Construct a &ldquo;bad&rdquo; trajectory: <strong>only step 2 is wrong</strong> (π taken as 3), steps 1 and 3 are both correct, and the final answer is wrong.</p>
<ul>
<li><strong>ORM (final outcome only)</strong>: on failure, every step&rsquo;s credit is 0—because <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn><mo>×</mo><mi>γ</mi><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">0 \times \gamma = 0</annotation></semantics></math></span>, after the zero signal propagates backward every step gets &ldquo;no signal&rdquo;; if you switch to &ldquo;-1 on failure&rdquo;, the correct search in step 1 receives a penalty of -0.857. Either the innocent get blamed, or everyone simply goes silent.</li>
<li><strong>PRM (each step scored independently)</strong>: correct steps get positive scores and wrong steps negative ones; the discriminability is 19 times that of ORM, and at the same number of steps the task success rate is about 30 percentage points higher. The cost is that every step has to be evaluated—which in real settings means annotation cost or training yet another scoring model.</li>
</ul>
<p>PRM is too expensive and ORM too coarse; the middle-ground routes of 2025–2026 fall into three categories by &ldquo;where the signal comes from&rdquo;, and all of them are essentially looking for a compromise between the two.</p>
<h2 id="the-credit-assignment-spectrum-three-routes-between-orm-and-prm">The Credit-Assignment Spectrum: Three Routes Between ORM and PRM</h2>
<p><strong>Route 1: compare actions from the same state (state-anchored).</strong> GiGPO&rsquo;s idea is that if multiple rollouts within a group converge on some state <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span>, you compare &ldquo;the subsequent returns of the different actions taken from the same situation&rdquo;, and the within-group relative score is the step advantage. Its refined version, HGPO, points out that the same page is not the same context (depending on whether a constraint was missed earlier, the same &ldquo;add to cart&rdquo; can mean something completely different), so it stratifies the comparison groups by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span>-step history and fuses the strata with weights. This route needs no extra annotation at all, but it depends on states being able to collide—web tasks are naturally friendly to it, while in open-ended dialogue collisions almost never happen.</p>
<p><strong>Route 2: redistribute the final reward back to each step (progress redistribution).</strong> SPA-RL trains a progress estimator under the constraint that the per-step contributions sum back to the final reward (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>R</mi><mo>^</mo></mover><mo>=</mo><msub><mo>∑</mo><mi>t</mi></msub><msub><mover accent="true"><mi>c</mi><mo>^</mo></mover><mi>t</mi></msub><mo>≈</mo><mi>R</mi></mrow><annotation encoding="application/x-tex">\hat R = \sum_t \hat c_t \approx R</annotation></semantics></math></span>): the contributions along a successful trajectory sum to nearly 1 and along a failed one to nearly 0, which forces the model to assign positive contribution to the steps that actually push the task forward. IGPO is cheaper: it directly uses &ldquo;the increase in the model&rsquo;s log probability of the ground-truth answer&rdquo; as the turn-level reward—if after turn <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span> the model is more confident in the correct answer (log-prob rises from -4.0 to -1.5), it records <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>r</mi><mrow><mi>I</mi><mi>G</mi></mrow></msup><mo>=</mo><mo>+</mo><mn>2.5</mn></mrow><annotation encoding="application/x-tex">r^{IG} = +2.5</annotation></semantics></math></span>; if the model was led astray, it records a negative value. What it yields is a ready-made turn-level scalar that can go straight into discounted accumulation, with no Monte Carlo value estimation needed; the cost is a dependence on high-quality reference answers, and questions with multiple valid answers get unfairly penalized.</p>
<p><strong>Route 3: change the optimization granularity (the step-aligned paradigm).</strong> StepPO and Turn-PPO take the most radical view: existing algorithms inherit RLHF&rsquo;s token-centric paradigm, but an agent&rsquo;s decisions are naturally turn-grained, so there is a granularity mismatch between token-level optimization and turn-level decisions. Turn-PPO uses PPO instead of GRPO for multi-turn training, on the grounds that trajectory-level within-group advantages have too much variance, and switches to turn-level advantage estimation—each turn has its own advantage, shared among the tokens within that turn. This route is orthogonal to all the methods above: the others change where <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> comes from; this one changes the basic unit of optimization.</p>
<p>Whichever route you take, multi-turn RL has one more problem, along a purely temporal dimension: <strong>the earlier the mistake, the greater the responsibility</strong> (if step 1 goes wrong, every later step unfolds on top of the error). Both the textbook and engineering implementations handle this with backward discounting: working back from the final outcome, the earlier the step, the larger the discount—for a 7-turn interaction in which only the last turn has reward 1.0 and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi><mo>=</mo><mn>0.9</mn></mrow><annotation encoding="application/x-tex">\gamma = 0.9</annotation></semantics></math></span>, the returns assigned to the turns are <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">[</mo><mn>0.531</mn><mo separator="true">,</mo><mn>0.590</mn><mo separator="true">,</mo><mn>0.656</mn><mo separator="true">,</mo><mn>0.729</mn><mo separator="true">,</mo><mn>0.810</mn><mo separator="true">,</mo><mn>0.900</mn><mo separator="true">,</mo><mn>1.000</mn><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">[0.531, 0.590, 0.656, 0.729, 0.810, 0.900, 1.000]</annotation></semantics></math></span>.</p>
<h2 id="where-the-grpogspo-family-sits-on-this-map">Where the GRPO/GSPO Family Sits on This Map</h2>
<p>This post settles the loose end left by the <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">GSPO post</a>
. GSPO&rsquo;s principle that &ldquo;the unit of optimization should match the unit of reward&rdquo; runs into real tension in multi-turn settings: GSPO&rsquo;s objective treats the entire response as a single action, which suits outcome-level rewards; multi-turn RL wants turn-level credit assignment, which amounts to demanding a granularity finer than the sequence. The GSPO-token variant that §4.3 of the GSPO paper designs for exactly this has a very telling formula:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>s</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>⋅</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><mrow><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">s_{i,t}(\theta) = \mathrm{sg}[s_i(\theta)] \cdot \frac{\pi_\theta(y_{i,t}|x,y_{i,&lt;t})}{\mathrm{sg}[\pi_\theta(y_{i,t}|x,y_{i,&lt;t})]}</annotation></semantics></math></span><p><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><mo>⋅</mo><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\mathrm{sg}[\cdot]</annotation></semantics></math></span> is stop-gradient (it takes only the value and passes no gradient back). The beauty of this construction: numerically, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">s_{i,t}</annotation></semantics></math></span> is identically equal to the sequence-level ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>, so the objective, the clipping condition, and the theoretical gradient are fully equivalent to GSPO&rsquo;s; yet the gradient expression now contains a &ldquo;per-token customizable advantage slot&rdquo;. Set all tokens of the same turn to the same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> and you get exactly turn-level credit assignment—this is a formal opening the algorithm layer leaves for multi-turn training, not an engineering hack.</p>
<p>The GRPO family takes the other, symmetric route: it leaves the mechanism of comparing multiple trajectories within a group untouched and outsources the construction of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>—whether you hand it trajectory-level comparisons (the default) or turn-level scores (ORM + discounting / PRM / IGPO / SALT), it multiplies them in all the same. So <strong>in implementation terms, &ldquo;turn-level reward in GRPO&rdquo; is not a change to the algorithm but a change to the advantage constructor + making sure the action mask is clean</strong>. The combination of the two routes at the paper level (GSPO-token × turn-level <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>) currently has no public ablation—a genuine gap.</p>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>the number of turns sets the tier; the algorithm can&rsquo;t.</strong> The textbook&rsquo;s selection guide tiers by turn count: for 3–5 turns, plain ORM/GRPO is enough (episodes are short, so sparsity isn&rsquo;t fatal); for 5–15 turns, bring in milestone rewards or annotation-free methods like SALT/GiGPO; beyond 15 turns, PRM or progress rewards plus tree search are a must. This tiering matters more than any single technique—when reward density can&rsquo;t sustain learning, switching to a stronger RL algorithm yields zero gain. The five-turn trajectory from the top of this post actually sits right on the boundary between the first and second tiers.</p>
<p>Second, <strong>the link between credit-assignment quality and entropy is more direct than you might think.</strong> Under ORM every step of a failed trajectory gets penalized, and the easiest countermeasure the model learns is to reduce behavioral diversity (take fewer actions, answer earlier)—in the multi-turn setting, entropy collapse is not a by-product but a direct consequence of wrong credit assignment. Conversely, fine-grained signals like PRM or IGPO let the model &ldquo;keep the right steps and fix only the wrong ones&rdquo;, which is what preserves exploration. In multi-turn training, watching the agent&rsquo;s turn-count distribution and tool-call diversity exposes credit-assignment problems earlier than watching the reward curve.</p>
<p>Third, <strong>four reward types stand between &ldquo;it runs&rdquo; and &ldquo;the credit is assigned correctly&rdquo;.</strong> The textbook divides agentic rewards into four categories: Outcome (is the final result correct), Format (can the action be parsed and executed), Cost (number of turns, API spend), and Process (does the step advance the task). When getting started, people usually implement only the first two, to make sure training runs, and Process is exactly what the credit-assignment layer needs to make dense. But many engineering setups also fold the Format penalty into the outcome signal and assign credit for both together—a turn with a format error and a turn with a reasoning error bear the same blame in the trajectory, and almost no public report teases this kind of conflation apart.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>There is no public ablation of how GSPO-token combined with turn-level <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> performs on multi-turn agentic tasks—whether Qwen has validated it internally is impossible to know;</li>
<li>The <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi></mrow><annotation encoding="application/x-tex">\gamma</annotation></semantics></math></span> of turn-level discounting has not been systematically swept in multi-turn LLM training (the textbook example uses 0.9, which comes from classical RL convention rather than an ablation);</li>
<li>The mix and isolation of the four reward types (outcome/format/cost/process) at the credit-assignment layer lack public controlled experiments—most reports give only final metrics, not the credit-assignment structure.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
: 22.2 Multi-turn RL formalization (AppWorld POMDP, action mask, the four reward types) and 22.3 Trajectory credit assignment (three signal layers, the ORM/PRM experiment, three middle-ground routes, turn-level discounting, selection tiers), in <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter22_agentic/formulation.md"><code>docs/chapter22_agentic/formulation.md</code></a>
 and <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter22_agentic/credit-assignment.md"><code>docs/chapter22_agentic/credit-assignment.md</code></a>
</li>
<li>GSPO paper: <a href="https://arxiv.org/abs/2507.18071">arXiv:2507.18071</a>
 §4.3 (the GSPO-token formula and the equivalence argument)</li>
<li>AppWorld: <a href="https://arxiv.org/abs/2407.18901">AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents</a>
; original ORM/PRM framework: <a href="https://arxiv.org/abs/2305.20050">Let&rsquo;s Verify Step by Step</a>
 (Lightman et al., 2023)</li>
<li>Middle-ground routes: SALT [2510.20022], GiGPO [2505.10978], HGPO [2602.22817], SPA-RL [2505.20732], IGPO [2510.14967], Turn-PPO [2512.17008], StepPO (the ID given in the textbook points to a different paper; the original could not be verified) (arXiv IDs checked against arXiv)</li>
<li>Previous posts: <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">The Ratio&rsquo;s Wrong Unit: GSPO Goes Sequence-Level</a>
, <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
</li>
</ul>
]]></content:encoded>
    </item>
  </channel>
</rss>
