Score Centering Stabilizes Off-policy Reinforcement Learning

Source facts

The preprint attributes instability in large-language-model reinforcement learning under training-inference mismatch to persistent drift between engines. It derives an additive score-centering correction that cancels the drift, and reports matching or exceeding importance-sampling methods across 0.6B to 30B parameter models under quantization and staleness conditions.

Analysis

The proposed correction targets an infrastructure-level source of RL instability without requiring complete elimination of training-inference mismatch. If replicated, it could reduce the cost of high-throughput rollouts while preserving off-policy training stability.

Uncertainty/Doubt

• This is an arXiv preprint; the reported scale comparisons and robustness claims were not independently reproduced here.