“The Model Trains Itself”: RSI and Three Loops
RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.
RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.
Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.
Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.
Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.
Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.
64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.
Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.
Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.
Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.
Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.