“The Model Trains Itself”: RSI and Three Loops

RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.

September 1, 2026 · 4 min · 1851 words · njx

A 7-Turn Failure: Assigning Blame in GRPO/GSPO

Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.

August 31, 2026 · 11 min · 2252 words · njx

One Vote or Ten: token-mean vs sample-mean

Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.

August 30, 2026 · 9 min · 1843 words · njx

Judging Truncated Answers: DAPO's Overlong Fix

Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.

August 30, 2026 · 9 min · 1770 words · njx

“Wrong? Write Longer”: What GRPO Normalizers Skew

Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.

August 29, 2026 · 10 min · 2061 words · njx

Samples per Prompt: What GRPO's Group Size Buys

64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.

August 29, 2026 · 11 min · 2324 words · njx

One Model, Two Probabilities: Cost of Mismatch

Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.

August 27, 2026 · 12 min · 2377 words · njx

The Ratio's Wrong Unit: GSPO Goes Sequence-Level

Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.

August 26, 2026 · 16 min · 3247 words · njx

Entropy Collapse: What DAPO's Clip-Higher Fixes

Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.

August 26, 2026 · 20 min · 4196 words · njx