Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Source

arXiv

arxiv.org/abs/2607.07508v1

Read the source

Source

Plain-Language Brief

The paper introduces Single-rollout Asynchronous Optimization (SAO), a new approach to reinforcement learning (RL) aimed at post-training large language models (LLMs). SAO addresses stability and off-policy challenges in asynchronous RL, improves generalization, and reduces off-policy effects by using one rollout per prompt. The method outperforms existing asynchronous RL systems like GRPO on agentic coding and reasoning benchmarks. It also excels in a simulated online learning setting where the model adapts to changing environments. SAO is currently being used in the RL pipeline for training the open GLM-5.2 model.

What the Document Says

  • The paper discusses Single-rollout Asynchronous Optimization (SAO) for reinforcement learning (RL) in post-training large language models (LLMs).
  • SAO addresses stability and off-policy challenges in asynchronous RL by replacing group-wise sampling with single-rollout sampling.
  • Single-rollout strategy is improved with practical value-model training designs.
  • A strict double-side token-level clipping strategy is introduced to improve optimization stability.
  • SAO outperforms GRPO and its variants on agentic coding and reasoning benchmarks such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench.
  • SAO demonstrates effectiveness in a simulated online learning setting.
  • SAO is used in the RL pipeline for training the open GLM-5.2 model.

Key Concepts Mentioned

  • Reinforcement Learning
  • Large Language Models
  • Asynchronous Reinforcement Learning
  • Single-rollout Asynchronous Optimization (SAO)
  • Group-wise Sampling
  • Value-model Training Designs
  • Optimization Stability
  • Off-policy Effects
  • Agentic Coding and Reasoning Benchmarks
  • SWE-Bench Verified
  • BeyondAIME
  • IMOAnswerBench
  • Simulated Online Learning Setting
  • GLM-5.2 Model

Claims Made by the Document

  • SAO outperforms existing asynchronous RL systems like GRPO.
  • Single-rollout RL is particularly effective in a simulated online learning setting.

Evidence Used

  • Consistent outperformance on agentic coding and reasoning benchmarks compared to GRPO.
  • Demonstrated effectiveness in a simulated online learning setting.

Methods / Approach

  • Single-rollout Asynchronous Optimization (SAO)

Limitations Stated by the Authors

  • Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The paper does not provide detailed information about the benchmarks used or the specific performance metrics.
  • The paper does not compare SAO to other methods outside of GRPO and its variants.

Open Questions

  • What specific performance metrics were used in the comparison between SAO and GRPO?
  • How does SAO compare to other methods beyond GRPO and its variants?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on the benefits and effectiveness of Single-rollout Asynchronous Optimization (SAO) for reinforcement learning in post-training large language models, but does not provide detailed information about the benchmarks used or the specific performance metrics.
  • The paper compares SAO to GRPO and its variants, but does not compare it to other methods.

Raw Abstract / Excerpt

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).