Hi, this is Jiaxing Ni. I write about harness and rollout systems, model architecture, mid-training and post-training, evals and benchmarks, reasoning RL and agentic RL, RSI, data synthesis, and the engineering work of turning ideas into systems. Posts are updated as I learn more.
“The Model Trains Itself”: RSI and Three Loops
RSI means different things to different people. Self-improvement splits into data, reward and system loops, each with its own conditions and failure modes.
A 7-Turn Failure: Assigning Blame in GRPO/GSPO
Multi-turn agentic RL has one reward per trajectory. How to split it across turns: the formalism, the credit-assignment spectrum, and GSPO-token’s opening.
One Vote or Ten: token-mean vs sample-mean
Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.
Judging Truncated Answers: DAPO's Overlong Fix
Scoring truncated responses as wrong injects reward noise into the advantage. DAPO either masks their loss or adds a linear penalty ramp before the length cap.
“Wrong? Write Longer”: What GRPO Normalizers Skew
Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.
Samples per Prompt: What GRPO's Group Size Buys
64, 8 or 2 samples per prompt? G is GRPO’s only baseline and its biggest compute knob. What it buys, and why dynamic sampling can stand in for a larger G.
Anatomy of a Minimal SFT Trainer: Every Decision
Why hand-write an SFT loop when TRL trains in one line? Seven decisions in a minimal trainer, checked against textbooks and production frameworks.
One Model, Two Probabilities: Cost of Mismatch
Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.
The Ratio's Wrong Unit: GSPO Goes Sequence-Level
Qwen says GRPO’s token-level importance ratio misuses importance sampling and breaks MoE training. GSPO lifts ratio, clipping and updates to sequence level.
Entropy Collapse: What DAPO's Clip-Higher Fixes
Naive GRPO collapses entropy fast. DAPO’s Clip-Higher only lifts the upper clip bound from 0.2 to 0.28, and the reason is an asymmetry built into PPO-Clip.