<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>self-improvement on Njx&#39;Log</title>
    <link>https://njx-njx.github.io/en/tags/self-improvement/</link>
    <description>Recent content in self-improvement on Njx&#39;Log</description>
    <image>
      <title>Njx&#39;Log</title>
      <url>https://njx-njx.github.io/blog-cover.jpg</url>
      <link>https://njx-njx.github.io/blog-cover.jpg</link>
    </image>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate>Tue, 27 Oct 2026 15:14:17 +0800</lastBuildDate>
    <atom:link href="https://njx-njx.github.io/en/tags/self-improvement/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>“The Model Trains Itself”: RSI and Three Loops</title>
      <link>https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/</link>
      <pubDate>Tue, 01 Sep 2026 12:30:00 +0800</pubDate>
      <guid>https://njx-njx.github.io/en/posts/rsi-self-improvement-loops/</guid>
      <description>RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.</description>
      <content:encoded><![CDATA[<p>The term RSI (recursive self-improvement) became hot again in 2025-2026, but the things it refers to often get lumped together. Starting from a practitioner&rsquo;s dichotomy (change the harness but not the weights vs. change the weights directly), this post breaks “self-improvement” into three levels—the data loop, the reward loop, and the system loop—which differ in their preconditions, how far they have actually paid off, and their failure modes.</p>
<blockquote>
<p>This post has a companion <a href="https://njx-njx.github.io/reports/rsi-self-improvement-loops/">interactive report</a>
 (in Chinese), with an interactive diagram of the three nested loops, a three-level exam matrix, and a scissors-gap diagram.</p>
</blockquote>
<h2 id="a-conflated-term">A Conflated Term</h2>
<p>&ldquo;RSI&rdquo; (Recursive Self-Improvement) became hot again in 2025–2026, but what people mean by it varies widely: some mean an iterative pipeline that filters rollouts into data and retrains on it, some mean a model rewriting its own training process, and some mean scaling up test-time compute. The term&rsquo;s degree of confusion is directly proportional to the allure of what it promises.</p>
<p>Let me start with a practitioner&rsquo;s framework. In a conversation, Ding Duo, head of post-training at Perplexity, split RSI and continual learning into two routes: <strong>one leaves the model parameters unchanged and optimizes the harness, the loop, and the solving process; the other updates the model weights directly</strong>. His judgment is that the former produces results faster, while the latter may bring deeper changes in capability; and the two routes share three hard problems—controlling cost, avoiding parameter contamination, and proving that what is continually learned is transferable capability. This post will keep coming back to these three problems.</p>
<p>Another common confusion is around the term test-time. Classic TTT (Test-Time Training, Sun et al. 2020) refers to taking gradient updates on the current input at inference time, whereas the &ldquo;test-time scaling&rdquo; popularized by s1 refers to lengthening the reasoning chain and adding budget, without touching the weights. When people in an LLM context talk about a &ldquo;new test-time paradigm&rdquo;, always ask which one they mean—the former actually changes the model; the latter just thinks more. What Ding Duo calls &ldquo;optimizing the harness/loop&rdquo; is in fact a third path: leave the gradients alone and change the scaffolding.</p>
<h2 id="level-one-the-data-loopmost-mature-and-least-like-rsi">Level One: The Data Loop—Most Mature, and Least Like RSI</h2>
<p>The first level is data circulating through the loop: model rollout → filter successful trajectories by verifiable outcomes → mix them into the next round of SFT/RL training → the model gets stronger → roll out again. LoopTool (<a href="https://arxiv.org/abs/2511.09148">arXiv:2511.09148</a>
) has exactly this shape (evaluation uncovers weaknesses → targeted data synthesis → retraining); a more complete industrial example is the WebSailor series (arXiv:2507.02592; the V2 version, 2509.13305, explicitly puts synthetic data + scalable RL in its title): use a strong model to generate browsing trajectories, train with outcome rewards, filter with self-critique, then automatically synthesize harder training tasks. DeepResearcher (2504.03160) reports a phenomenon worth remembering: in this kind of closed-loop training, behaviors such as pre-search planning, information layering, and cross-validation <strong>are not explicitly encouraged by any reward term; they are by-products of RL optimization</strong>.</p>
<p>This level is already industrialized, but strictly speaking it is not &ldquo;self&rdquo;-improvement—the choices in the loop are made by a filter designed by engineers, and the model is just a data source. Its ceiling is also clear: the filtering criteria themselves are hackable (inefficient trajectories that &ldquo;happened to succeed&rdquo; get kept), and the narrower the data distribution, the faster the loop turns, and the faster the model converges itself toward whatever the current verifier likes.</p>
<h2 id="level-two-the-reward-looprlsvr-aims-to-extend-the-self-verification-dividend-to-open-ended-tasks">Level Two: The Reward Loop—RLSVR Aims to Extend the Self-Verification Dividend to Open-Ended Tasks</h2>
<p>The second level is reward circulating through the loop. RLVR&rsquo;s success over the past two years rests on &ldquo;math and code have deterministic verifiers&rdquo;; open-ended tasks (writing, research, dialogue) have no verifier and can only rely on preference models or LLM judges, which brings a triple tax: evaluation bias, the judge-capability bottleneck, and inference cost. RLSVR (arXiv:2607.23802) borrows the concept of a &ldquo;pretext task&rdquo; from self-supervised learning: <strong>apply a task transformation to the open-ended task to turn it into a self-verifiable form</strong>, so that the reward once again comes from the task itself rather than from an external judge.</p>
<p>The conditions for this idea to hold are worth laying out: it requires that the task admit some fidelity-preserving transformation (getting the transformed task right ≈ doing the original task well), and that the verifier be deterministic in the transformed space. The &ldquo;task transformation induces self-verifiable rewards&rdquo; in the paper&rsquo;s title refers to exactly this inducing step. If it holds, open-ended tasks can also tap into the dividends of RLVR&rsquo;s industrialization; if it doesn&rsquo;t—if the transformation loses too much—the self-verifiable reward is optimizing a proxy metric that has come apart from the original task.</p>
<p>What this means for the loops: the reward loop is the most critical of the three levels for RSI. The data loop&rsquo;s filter and the system loop&rsquo;s evaluator both have to answer &ldquo;what counts as getting better&rdquo;, and self-verifiable reward is currently the only answer that does not depend on a continuous supply from humans.</p>
<h2 id="level-three-the-system-looprsi-proper-stuck-on-oversight">Level Three: The System Loop—RSI Proper, Stuck on Oversight</h2>
<p>Only the third level is RSI proper: the model (or what it produces) improves the things used to improve it—better assistants, better data, better evaluations—forming a recursion. Leike et al.&rsquo;s 2018 research agenda for reward modeling (arXiv:1811.07871) already has a section explicitly discussing recursive reward modeling: use AI to help humans evaluate AI outputs, so that evaluation capability grows along with whatever is being evaluated.</p>
<p>The prerequisite for this level is scalable oversight (2211.03540, cited in Chapter 30 of the hands-on-modern-rl textbook as a core direction in alignment): once model capability exceeds human evaluation capability, who verifies that an improvement is real? Every round of the loop makes selections, and once the selection criteria are hackable, <strong>the faster the loop turns, the faster it collapses</strong>—a small-scale version is already visible in the data loop (metric inflation), and the system loop amplifies it to the entire pipeline. This is also why &ldquo;proving that what is continually learned is transferable capability&rdquo; is on Ding Duo&rsquo;s list of hard problems: improvement inside the loop is often &ldquo;getting stronger on this loop&rsquo;s evaluations&rdquo;, which does not necessarily hold outside the loop.</p>
<h2 id="the-three-levels-share-one-exam">The Three Levels Share One Exam</h2>
<p>Putting the three levels together, each of Ding Duo&rsquo;s three hard problems maps onto concrete mechanisms:</p>
<ul>
<li><strong>Cost</strong>: the data loop&rsquo;s cost is rollout and filtering (every iteration samples in full; we did the rollout = prompt × G math earlier in the series); the reward loop&rsquo;s cost is constructing and maintaining the task transformations and verifiers; the system loop&rsquo;s cost is basically impossible to estimate for now—because no instance has yet run the full pipeline end to end.</li>
<li><strong>Parameter contamination</strong>: only level three is involved. Continual gradient updates let new knowledge interfere with old capabilities, and the contamination accumulates with the number of iterations—this is one reason the &ldquo;don&rsquo;t change the weights&rdquo; route is currently underrated: optimizing the harness and loop doesn&rsquo;t contaminate parameters, and the gains are immediate; engineering work like DeepSeek V3.2 (self-added verifiers, switching the sampler to mHC) is this route paying off.</li>
<li><strong>Transferability</strong>: none of the three levels has a good evaluation protocol. Evaluation inside a loop is contaminated by the loop itself (the training target and the evaluation target share the same source), and cross-loop evaluation is too expensive.</li>
</ul>
<h2 id="my-three-takes">My Three Takes</h2>
<p>First, <strong>the &ldquo;don&rsquo;t change the weights&rdquo; route is currently the main way RSI actually pays off, and theorists underrate it</strong>. Scaffold optimization (tools, context engineering, verifier internalization, sampling procedures) doesn&rsquo;t change parameters, doesn&rsquo;t contaminate weights, and can be rolled back, and at the current stage its return per unit of compute is higher than that of changing the weights directly. Expelling it from the ranks of &ldquo;self-improvement&rdquo; is a misallocation of resources caused by terminological purism.</p>
<p>Second, <strong>the three levels are bottlenecked in different places</strong>. The data loop is stuck on the hackability of its filter; the reward loop on the fidelity of the task transformation; the system loop on oversight (without a solution to scalable oversight, recursion can only be recursive self-deception). Discuss &ldquo;RSI progress&rdquo; without specifying the level, and the conclusions are bound to be vague.</p>
<p>Third, <strong>loop speed is a double-edged sword, and an amplifier of safety problems</strong>. Every paper on self-improvement talks about convergence speed, and few talk about runaway speed at the same time; yet the two share the same structure—the loop compounds &ldquo;small selection biases&rdquo;. For self-improving systems, what should be monitored is the <strong>scissors gap</strong> between in-loop and out-of-loop metrics, not any single metric.</p>
<h2 id="open-questions">Open Questions</h2>
<ul>
<li>Whether there is a systematic way to construct task transformations for open-ended tasks (rather than hand-tuning them task by task); RLSVR only provides the paradigm;</li>
<li>Where the capability ceiling of the &ldquo;don&rsquo;t change the weights&rdquo; route lies—whether there are capabilities that can only be obtained by changing the weights; there is no clean evidence yet;</li>
<li>An evaluation protocol for cross-loop transferability: who evaluates, with what data, and how to keep the evaluation from being contaminated by any one of the loops.</li>
</ul>
<h2 id="references">References</h2>
<ul>
<li>Notes on a conversation with Ding Duo: 《在模型持续进步的时代，创业者应该站在哪里》§20 (the two routes for RSI and continual learning, and the three hard problems)</li>
<li>RLSVR: <a href="https://arxiv.org/abs/2607.23802">arXiv:2607.23802</a>
 (abstract and core claims)</li>
<li>WebSailor: <a href="https://arxiv.org/abs/2507.02592">arXiv:2507.02592</a>
, <a href="https://arxiv.org/abs/2509.13305">V2 2509.13305</a>
 (titles and positioning within the series); DeepResearcher <a href="https://arxiv.org/abs/2504.03160">arXiv:2504.03160</a>
</li>
<li>Recursive reward modeling: Leike et al. <a href="https://arxiv.org/abs/1811.07871">arXiv:1811.07871</a>
; Scalable Oversight <a href="https://arxiv.org/abs/2211.03540">arXiv:2211.03540</a>
 (hands-on-modern-rl textbook: <a href="https://github.com/walkinglabs/hands-on-modern-rl/blob/main/docs/chapter30_alignment_failures/scaling-and-defenses.md"><code>docs/chapter30_alignment_failures/scaling-and-defenses.md</code></a>
)</li>
<li>Classic Test-Time Training: Sun et al. <a href="https://arxiv.org/abs/1909.13231">arXiv:1909.13231</a>
; test-time scaling: s1 <a href="https://arxiv.org/abs/2501.19393">arXiv:2501.19393</a>
</li>
<li>Previous post: <a href="https://njx-njx.github.io/en/posts/turn-level-reward-credit-assignment/">Who&rsquo;s to Blame When a 7-Turn Interaction Fails: Reward Credit Assignment for Multi-Turn Trajectories</a>
 (previous post: within-episode credit assignment; this post: cross-episode loops)</li>
</ul>
]]></content:encoded>
    </item>
  </channel>
</rss>
