Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

OR Else: A Differentiable Trust Region for Policy Optimization

Source

arXiv

arxiv.org/abs/2607.18163v1

Read the source

Source

Plain-Language Brief

The paper compares PPO (Proximal Policy Optimization) and GRPO (Group Regularized Policy Optimization) with two variations using a differentiable trust region for policy optimization called Output Reset (OR). The study uses a large language model, Llama-3.2-1B-Instruct, on the Anthropic hh-rlhf dataset. PPO-OR and GRPO-OR perform better than their clipped counterparts under Generalized Advantage Estimation (GAE) and group-relative advantages respectively, showing improvements in certain metrics like mean final training-time reward-model score, observed spread, and terminal OR residual. However, the larger rollout-to-current log-ratio displacement is not consistently reduced by OR, and the reward effects differ between the compared methods.

What the Document Says

  • PPO-OR and GRPO-OR use Output Reset (OR) instead of clipped policy term in rollout-relative token log-ratio space
  • PPO-clip is compared with PPO-OR under Generalized Advantage Estimation (GAE)
  • GRPO is compared with GRPO-OR under group-relative advantages
  • The experiments are conducted using Llama-3.2-1B-Instruct on Anthropic hh-rlhf with one shared reward model and three seeds per method
  • PPO-OR has a higher mean final training-time reward-model score under GAE than PPO-clip
  • GRPO-OR does not have a higher mean score but shows smaller observed spread, near-zero terminal OR residual, and declining overshoot fraction under group-relative advantages
  • Both group-relative methods exhibit larger rollout-to-current log-ratio displacement than GAE methods
  • OR does not consistently reduce the rollout-to-current log-ratio displacement

Key Concepts Mentioned

  • PPO
  • GRPO
  • Output Reset (OR)
  • Generalized Advantage Estimation (GAE)
  • rollout-relative token log-ratio space
  • Llama-3.2-1B-Instruct
  • Anthropic hh-rlhf
  • reward model

Claims Made by the Document

  • PPO-OR has a higher mean final training-time reward-model score under GAE than PPO-clip
  • GRPO-OR shows improvements in certain metrics like observed spread, terminal OR residual, and overshoot fraction under group-relative advantages

Evidence Used

  • mean final training-time reward-model score $0.305$ higher for PPO-OR than PPO-clip under GAE
  • smaller observed spread, near-zero terminal OR residual, and declining overshoot fraction for GRPO-OR under group-relative advantages

Methods / Approach

  • PPO
  • GRPO
  • Output Reset (OR)

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

PPO and the GRPO baseline studied here use clipped surrogate objectives whose favorable-direction saturation introduces an abrupt change in the scalar objective's derivative. We ask whether Output Reset (OR), a smooth one-sided saturation rule, offers a useful alternative for large language model post-training. PPO-OR and GRPO-OR replace the clipped policy term with an OR squared-margin loss in rollout-relative token log-ratio space; the advantage sign determines the update direction, and a token contributes zero direct OR residual after crossing the favorable margin. We compare PPO-clip with PPO-OR under generalized advantage estimation (GAE), and GRPO with GRPO-OR under group-relative advantages, using \texttt{Llama-3.2-1B-Instruct} on Anthropic \texttt{hh-rlhf} with one shared reward model and three seeds per method. Under GAE, PPO-OR has a mean final training-time reward-model score $0.305$ higher than PPO-clip, with a larger observed across-seed spread. Under group-relative advantages, GRPO-OR does not have a higher mean score, but shows a smaller observed spread, a near-zero terminal OR residual, and a declining overshoot fraction, while the matched GRPO clipped-objective trace remains variable. Both group-relative methods exhibit substantially larger rollout-to-current log-ratio displacement than the GAE methods, and OR does not consistently reduce it. Thus, OR changes optimization behavior in both matched comparisons, but the observed reward effect differs between them. At $G=2$, the GRPO-OR diagnostics do not translate into a reward-score gain. Whether larger groups change this outcome remains open. The reported scores are training-time reward-model measurements, not held-out human-preference performance.