One Vote or Ten: token-mean vs sample-mean
Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.
Does a 5000-token answer outweigh a 500-token one? Sample-mean gives each response a vote, token-mean each token, and switching has a hidden cost.
Part of R1-Zero’s growing length may be the optimizer’s own bias: 1/|y| favors long wrong answers and dividing by std overweights extreme questions.