One Model, Two Probabilities: Cost of Mismatch
Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.
Rollout and training engines compute different probabilities for one model, so on-policy RL quietly turns off-policy. Where it comes from and how to fix it.