<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>credit-assignment on Njx&#39;Log</title>
    <link>https://njx-njx.github.io/en/tags/credit-assignment/</link>
    <description>Recent content in credit-assignment on Njx&#39;Log</description>
    <image>
      <title>Njx&#39;Log</title>
      <url>https://njx-njx.github.io/blog-cover.jpg</url>
      <link>https://njx-njx.github.io/blog-cover.jpg</link>
    </image>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate>Tue, 27 Oct 2026 15:14:17 +0800</lastBuildDate>
    <atom:link href="https://njx-njx.github.io/en/tags/credit-assignment/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>A 7-Turn Failure: Assigning Blame in GRPO/GSPO</title>
      <link>https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/</link>
      <pubDate>Mon, 31 Aug 2026 12:55:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/</guid>
      <description>Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token&amp;rsquo;s opening.</description>
      <content:encoded><![CDATA[<p>In single-turn RL the whole sequence shares one advantage, and nobody has ever lost sleep over it; in multi-turn agentic RL, the same trajectory can get the turn-1 search right, send the turn-2 query off course, and end with a wrong answer—so how to split a single trajectory reward becomes a real design decision. This post unpacks the formalism (action mask, trajectory probability), the credit-assignment spectrum (ORM/PRM/turn discounting/three middle-ground routes), and the opening GSPO-token leaves for multi-turn training.</p>
<blockquote>
<p>This post comes with a companion <a href="https://njx-njx.github.io/reports/turn-level-reward-credit-assignment/">interactive report</a>
 (in Chinese), featuring a toggleable ORM/PRM credit-assignment comparison, backward discounting with a γ slider, and a breakdown of the GSPO-token formula.</p>
</blockquote>
<h2 id="a-problem-that-doesnt-exist-in-single-turn-training">A Problem That Doesn&rsquo;t Exist in Single-Turn Training</h2>
<p>Consider a five-turn trajectory: turn 1&rsquo;s search hits the key evidence; turn 2 drifts off course while rewriting the query and brings back irrelevant content; turn 3 corrects course and gets halfway back; turn 4 makes a calculation error; turn 5 gives a wrong answer. The environment returns just one number: failure, reward = 0.</p>
<p>Single-turn RL doesn&rsquo;t have this problem—the entire response shares one advantage, and &ldquo;all right or all wrong&rdquo; is taken for granted. In the multi-turn setting this crude logic breaks down immediately: turn 1 clearly did the right thing, so why should it be penalized along with turn 4? On top of that there is a further engineering layer: the trajectory mixes two kinds of tokens, those the model generated itself (thinking, tool-call arguments, the final answer) and those returned by the environment (search results, code output), and the latter should never enter the gradient at all. So &ldquo;how does turn-level reward get into GRPO/GSPO&rdquo; is really three questions rolled into one: <strong>which tokens count as actions, how much advantage each turn gets, and which tokens that advantage gets multiplied onto</strong>. This post takes them apart in that order.</p>
<h2 id="first-pin-down-the-object-trajectory-probability-and-the-action-mask">First, Pin Down the Object: Trajectory Probability and the Action Mask</h2>
<p>In the formalization of multi-turn trajectories (the textbook adopts the POMDP formulation from the AppWorld paper, arXiv:2504.11536), the full state is <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>z</mi><mi>t</mi></msub><mo>=</mo><mo stretchy="false">[</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>c</mi><mo separator="true">,</mo><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi></mrow></msub><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">z_t = [s_0, c, x_{1:t}]</annotation></semantics></math></span>: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>0</mn></msub></mrow><annotation encoding="application/x-tex">s_0</annotation></semantics></math></span> is the hidden initial environment state (database snapshot, REPL state), <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>c</mi></mrow><annotation encoding="application/x-tex">c</annotation></semantics></math></span> is the task context, and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">x_{1:t}</annotation></semantics></math></span> is the entire token history so far—both what the model generated and what the environment returned. The model sees only the text part, which is why it is called partially observable.</p>
<p>The chain-rule factorization of the trajectory probability has one key detail:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>ρ</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><mi>x</mi><mo>∣</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>c</mi><mo stretchy="false">)</mo><mo>=</mo><mi mathvariant="double-struck">I</mi><mo stretchy="false">(</mo><msub><mi>s</mi><mn>0</mn></msub><mo separator="true">,</mo><mi>x</mi><mo stretchy="false">)</mo><munder><mo>∏</mo><mrow><mi>t</mi><mo>∈</mo><mi>a</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow></munder><msub><mi>p</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>x</mi><mi>t</mi></msub><mo>∣</mo><mi>c</mi><mo separator="true">,</mo><msub><mi>x</mi><mrow><mn>1</mn><mo>:</mo><mi>t</mi><mo>−</mo><mn>1</mn></mrow></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\rho_\theta(x \mid s_0, c) = \mathbb{I}(s_0, x) \prod_{t \in a(x)} p_\theta(x_t \mid c, x_{1:t-1})</annotation></semantics></math></span><p>The product runs <strong>only over <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>a</mi><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">a(x)</annotation></semantics></math></span>, the set of positions of model-generated tokens</strong>; the observation tokens returned by the environment are determined by the environment dynamics and are not part of the product. Translated into the loss, this is the action mask: a 0/1 vector the same length as the trajectory, with model tokens marked 1 and the prompt and tool returns marked 0. SearchR1&rsquo;s implementation makes this very explicit: the contents of <code>&lt;think&gt;</code> and <code>&lt;search&gt;</code> take part in training, while the <code>&lt;information&gt;</code> segments returned by the retriever are masked out entirely. Letting environment tokens into the gradient as well would amount to forcing the model to &ldquo;learn to predict which web pages the search engine returns&rdquo;, contaminating the policy gradient.</p>
<p>This sets the skeleton of this post&rsquo;s answer: turn-level reward enters training <strong>not by weighting environment tokens, but by constructing a separate advantage for each turn&rsquo;s model tokens</strong>.</p>
<h2 id="three-layers-of-signal-how-reward-travels-from-trajectory-to-weights">Three Layers of Signal: How Reward Travels from Trajectory to Weights</h2>
<p>The textbook splits the signal flow into three layers, and this layering is the map for understanding the whole spectrum of approaches:</p>
<ol>
<li><strong>Trajectory reward</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>R</mi><mo stretchy="false">(</mo><msub><mi>τ</mi><mi>i</mi></msub><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">R(\tau_i)</annotation></semantics></math></span>: the raw scalar given by the environment—&ldquo;did this episode succeed or not&rdquo;;</li>
<li><strong>Step advantage</strong> <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>: breaks the final outcome down into per-turn scores, answering &ldquo;should turn <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span> be rewarded&rdquo;—this is the main battleground of credit assignment;</li>
<li><strong>Token gradient</strong>: <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> is multiplied onto the log-probs of that turn&rsquo;s action tokens, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi mathvariant="normal">Δ</mi><mi>θ</mi><mo>=</mo><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><msub><mo>∑</mo><mrow><mi>k</mi><mo>∈</mo><msub><mi>a</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow></msub><mi mathvariant="normal">∇</mi><mi>log</mi><mo>⁡</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mi>k</mi></msub><mo>∣</mo><mo>⋅</mo><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">\Delta\theta = A_{i,t} \sum_{k \in a_{i,t}} \nabla \log \pi_\theta(y_k \mid \cdot)</annotation></semantics></math></span>.</li>
</ol>
<p>What &ldquo;putting turn-level reward into GRPO&rdquo; actually comes down to is layer 2: GRPO&rsquo;s within-group comparison naturally works at the trajectory level (ranking multiple trajectories for the same problem against each other), so to get down to the turn level you have to define <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> yourself and feed it in.</p>
<h2 id="why-the-naive-approach-fails-a-controlled-experiment">Why the Naive Approach Fails: A Controlled Experiment</h2>
<p>The textbook&rsquo;s Mini Agent Loop experiment gives the most convincing set of numbers. The task: look up Earth&rsquo;s radius → compute the equatorial circumference → verify → answer. Construct a &ldquo;bad&rdquo; trajectory: <strong>only step 2 is wrong</strong> (π taken as 3), steps 1 and 3 are both correct, and the final answer is wrong.</p>
<ul>
<li><strong>ORM (final outcome only)</strong>: on failure, every step&rsquo;s credit is 0—because <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mn>0</mn><mo>×</mo><mi>γ</mi><mo>=</mo><mn>0</mn></mrow><annotation encoding="application/x-tex">0 \times \gamma = 0</annotation></semantics></math></span>, after the zero signal propagates backward every step gets &ldquo;no signal&rdquo;; if you switch to &ldquo;-1 on failure&rdquo;, the correct search in step 1 receives a penalty of -0.857. Either the innocent get blamed, or everyone simply goes silent.</li>
<li><strong>PRM (each step scored independently)</strong>: correct steps get positive scores and wrong steps negative ones; the discriminability is 19 times that of ORM, and at the same number of steps the task success rate is about 30 percentage points higher. The cost is that every step has to be evaluated—which in real settings means annotation cost or training yet another scoring model.</li>
</ul>
<p>PRM is too expensive and ORM too coarse; the middle-ground routes of 2025–2026 fall into three categories by &ldquo;where the signal comes from&rdquo;, and all of them are essentially looking for a compromise between the two.</p>
<h2 id="the-credit-assignment-spectrum-three-routes-between-orm-and-prm">The Credit-Assignment Spectrum: Three Routes Between ORM and PRM</h2>
<p><strong>Route 1: compare actions from the same state (state-anchored).</strong> GiGPO&rsquo;s idea is that if multiple rollouts within a group converge on some state <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>s</mi></mrow><annotation encoding="application/x-tex">s</annotation></semantics></math></span>, you compare &ldquo;the subsequent returns of the different actions taken from the same situation&rdquo;, and the within-group relative score is the step advantage. Its refined version, HGPO, points out that the same page is not the same context (depending on whether a constraint was missed earlier, the same &ldquo;add to cart&rdquo; can mean something completely different), so it stratifies the comparison groups by <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>k</mi></mrow><annotation encoding="application/x-tex">k</annotation></semantics></math></span>-step history and fuses the strata with weights. This route needs no extra annotation at all, but it depends on states being able to collide—web tasks are naturally friendly to it, while in open-ended dialogue collisions almost never happen.</p>
<p><strong>Route 2: redistribute the final reward back to each step (progress redistribution).</strong> SPA-RL trains a progress estimator under the constraint that the per-step contributions sum back to the final reward (<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>R</mi><mo>^</mo></mover><mo>=</mo><msub><mo>∑</mo><mi>t</mi></msub><msub><mover accent="true"><mi>c</mi><mo>^</mo></mover><mi>t</mi></msub><mo>≈</mo><mi>R</mi></mrow><annotation encoding="application/x-tex">\hat R = \sum_t \hat c_t \approx R</annotation></semantics></math></span>): the contributions along a successful trajectory sum to nearly 1 and along a failed one to nearly 0, which forces the model to assign positive contribution to the steps that actually push the task forward. IGPO is cheaper: it directly uses &ldquo;the increase in the model&rsquo;s log probability of the ground-truth answer&rdquo; as the turn-level reward—if after turn <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>t</mi></mrow><annotation encoding="application/x-tex">t</annotation></semantics></math></span> the model is more confident in the correct answer (log-prob rises from -4.0 to -1.5), it records <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msup><mi>r</mi><mrow><mi>I</mi><mi>G</mi></mrow></msup><mo>=</mo><mo>+</mo><mn>2.5</mn></mrow><annotation encoding="application/x-tex">r^{IG} = +2.5</annotation></semantics></math></span>; if the model was led astray, it records a negative value. What it yields is a ready-made turn-level scalar that can go straight into discounted accumulation, with no Monte Carlo value estimation needed; the cost is a dependence on high-quality reference answers, and questions with multiple valid answers get unfairly penalized.</p>
<p><strong>Route 3: change the optimization granularity (the step-aligned paradigm).</strong> StepPO and Turn-PPO take the most radical view: existing algorithms inherit RLHF&rsquo;s token-centric paradigm, but an agent&rsquo;s decisions are naturally turn-grained, so there is a granularity mismatch between token-level optimization and turn-level decisions. Turn-PPO uses PPO instead of GRPO for multi-turn training, on the grounds that trajectory-level within-group advantages have too much variance, and switches to turn-level advantage estimation—each turn has its own advantage, shared among the tokens within that turn. This route is orthogonal to all the methods above: the others change where <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> comes from; this one changes the basic unit of optimization.</p>
<p>Whichever route you take, multi-turn RL has one more problem, along a purely temporal dimension: <strong>the earlier the mistake, the greater the responsibility</strong> (if step 1 goes wrong, every later step unfolds on top of the error). Both the textbook and engineering implementations handle this with backward discounting: working back from the final outcome, the earlier the step, the larger the discount—for a 7-turn interaction in which only the last turn has reward 1.0 and <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi><mo>=</mo><mn>0.9</mn></mrow><annotation encoding="application/x-tex">\gamma = 0.9</annotation></semantics></math></span>, the returns assigned to the turns are <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo stretchy="false">[</mo><mn>0.531</mn><mo separator="true">,</mo><mn>0.590</mn><mo separator="true">,</mo><mn>0.656</mn><mo separator="true">,</mo><mn>0.729</mn><mo separator="true">,</mo><mn>0.810</mn><mo separator="true">,</mo><mn>0.900</mn><mo separator="true">,</mo><mn>1.000</mn><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">[0.531, 0.590, 0.656, 0.729, 0.810, 0.900, 1.000]</annotation></semantics></math></span>.</p>
<h2 id="where-the-grpogspo-family-sits-on-this-map">Where the GRPO/GSPO Family Sits on This Map</h2>
<p>This post settles the loose end left by the <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">GSPO post</a>
. GSPO&rsquo;s principle that &ldquo;the unit of optimization should match the unit of reward&rdquo; runs into real tension in multi-turn settings: GSPO&rsquo;s objective treats the entire response as a single action, which suits outcome-level rewards; multi-turn RL wants turn-level credit assignment, which amounts to demanding a granularity finer than the sequence. The GSPO-token variant that §4.3 of the GSPO paper designs for exactly this has a very telling formula:</p>
<span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><msub><mi>s</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><msub><mi>s</mi><mi>i</mi></msub><mo stretchy="false">(</mo><mi>θ</mi><mo stretchy="false">)</mo><mo stretchy="false">]</mo><mo>⋅</mo><mfrac><mrow><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo></mrow><mrow><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><msub><mi>π</mi><mi>θ</mi></msub><mo stretchy="false">(</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub><mi mathvariant="normal">∣</mi><mi>x</mi><mo separator="true">,</mo><msub><mi>y</mi><mrow><mi>i</mi><mo separator="true">,</mo><mo>&lt;</mo><mi>t</mi></mrow></msub><mo stretchy="false">)</mo><mo stretchy="false">]</mo></mrow></mfrac></mrow><annotation encoding="application/x-tex">s_{i,t}(\theta) = \mathrm{sg}[s_i(\theta)] \cdot \frac{\pi_\theta(y_{i,t}|x,y_{i,&lt;t})}{\mathrm{sg}[\pi_\theta(y_{i,t}|x,y_{i,&lt;t})]}</annotation></semantics></math></span><p><span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mrow><mi mathvariant="normal">s</mi><mi mathvariant="normal">g</mi></mrow><mo stretchy="false">[</mo><mo>⋅</mo><mo stretchy="false">]</mo></mrow><annotation encoding="application/x-tex">\mathrm{sg}[\cdot]</annotation></semantics></math></span> is stop-gradient (it takes only the value and passes no gradient back). The beauty of this construction: numerically, <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">s_{i,t}</annotation></semantics></math></span> is identically equal to the sequence-level ratio <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mi>i</mi></msub></mrow><annotation encoding="application/x-tex">s_i</annotation></semantics></math></span>, so the objective, the clipping condition, and the theoretical gradient are fully equivalent to GSPO&rsquo;s; yet the gradient expression now contains a &ldquo;per-token customizable advantage slot&rdquo;. Set all tokens of the same turn to the same <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> and you get exactly turn-level credit assignment—this is a formal opening the algorithm layer leaves for multi-turn training, not an engineering hack.</p>
<p>The GRPO family takes the other, symmetric route: it leaves the mechanism of comparing multiple trajectories within a group untouched and outsources the construction of <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>—whether you hand it trajectory-level comparisons (the default) or turn-level scores (ORM + discounting / PRM / IGPO / SALT), it multiplies them in all the same. So <strong>in implementation terms, &ldquo;turn-level reward in GRPO&rdquo; is not a change to the algorithm but a change to the advantage constructor + making sure the action mask is clean</strong>. The combination of the two routes at the paper level (GSPO-token × turn-level <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span>) currently has no public ablation—a genuine gap.</p>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>the number of turns sets the tier; the algorithm can&rsquo;t.</strong> The textbook&rsquo;s selection guide tiers by turn count: for 3–5 turns, plain ORM/GRPO is enough (episodes are short, so sparsity isn&rsquo;t fatal); for 5–15 turns, bring in milestone rewards or annotation-free methods like SALT/GiGPO; beyond 15 turns, PRM or progress rewards plus tree search are a must. This tiering matters more than any single technique—when reward density can&rsquo;t sustain learning, switching to a stronger RL algorithm yields zero gain. The five-turn trajectory from the top of this post actually sits right on the boundary between the first and second tiers.</p>
<p>Second, <strong>the link between credit-assignment quality and entropy is more direct than you might think.</strong> Under ORM every step of a failed trajectory gets penalized, and the easiest countermeasure the model learns is to reduce behavioral diversity (take fewer actions, answer earlier)—in the multi-turn setting, entropy collapse is not a by-product but a direct consequence of wrong credit assignment. Conversely, fine-grained signals like PRM or IGPO let the model &ldquo;keep the right steps and fix only the wrong ones&rdquo;, which is what preserves exploration. In multi-turn training, watching the agent&rsquo;s turn-count distribution and tool-call diversity exposes credit-assignment problems earlier than watching the reward curve.</p>
<p>Third, <strong>four reward types stand between &ldquo;it runs&rdquo; and &ldquo;the credit is assigned correctly&rdquo;.</strong> The textbook divides agentic rewards into four categories: Outcome (is the final result correct), Format (can the action be parsed and executed), Cost (number of turns, API spend), and Process (does the step advance the task). When getting started, people usually implement only the first two, to make sure training runs, and Process is exactly what the credit-assignment layer needs to make dense. But many engineering setups also fold the Format penalty into the outcome signal and assign credit for both together—a turn with a format error and a turn with a reasoning error bear the same blame in the trajectory, and almost no public report teases this kind of conflation apart.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>There is no public ablation of how GSPO-token combined with turn-level <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>A</mi><mrow><mi>i</mi><mo separator="true">,</mo><mi>t</mi></mrow></msub></mrow><annotation encoding="application/x-tex">A_{i,t}</annotation></semantics></math></span> performs on multi-turn agentic tasks—whether Qwen has validated it internally is impossible to know;</li>
<li>The <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>γ</mi></mrow><annotation encoding="application/x-tex">\gamma</annotation></semantics></math></span> of turn-level discounting has not been systematically swept in multi-turn LLM training (the textbook example uses 0.9, which comes from classical RL convention rather than an ablation);</li>
<li>The mix and isolation of the four reward types (outcome/format/cost/process) at the credit-assignment layer lack public controlled experiments—most reports give only final metrics, not the credit-assignment structure.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Textbook <a href="https://github.com/walkinglabs/hands-on-modern-rl">hands-on-modern-rl</a>
: 22.2 Multi-turn RL formalization (AppWorld POMDP, action mask, the four reward types) and 22.3 Trajectory credit assignment (three signal layers, the ORM/PRM experiment, three middle-ground routes, turn-level discounting, selection tiers), in <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter22_agentic/formulation.md"><code>docs/chapter22_agentic/formulation.md</code></a>
 and <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter22_agentic/credit-assignment.md"><code>docs/chapter22_agentic/credit-assignment.md</code></a>
</li>
<li>GSPO paper: <a href="https://arxiv.org/abs/2507.18071">arXiv:2507.18071</a>
 §4.3 (the GSPO-token formula and the equivalence argument)</li>
<li>AppWorld: <a href="https://arxiv.org/abs/2407.18901">AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents</a>
; original ORM/PRM framework: <a href="https://arxiv.org/abs/2305.20050">Let&rsquo;s Verify Step by Step</a>
 (Lightman et al., 2023)</li>
<li>Middle-ground routes: SALT [2510.20022], GiGPO [2505.10978], HGPO [2602.22817], SPA-RL [2505.20732], IGPO [2510.14967], Turn-PPO [2512.17008], StepPO (the ID given in the textbook points to a different paper; the original could not be verified) (arXiv IDs checked against arXiv)</li>
<li>Previous posts: <a href="https://njx-njx.github.io/en/posts/gspo-sequence-level-importance-sampling/">The Ratio&rsquo;s Wrong Unit: GSPO Goes Sequence-Level</a>
, <a href="https://njx-njx.github.io/en/posts/dapo-clip-higher-entropy-collapse/">Entropy Collapse: What DAPO&rsquo;s Clip-Higher Fixes</a>
</li>
</ul>
]]></content:encoded>
    </item>
  </channel>
</rss>
