SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Dayang Liang, Lang Feng, Bo An, Yunlong Liu
• Venue: arXiv: cs.AI
• Published: 2026-08-20
• Collected: 2026-08-21
• URL: http://arxiv.org/abs/2608.19842v1
• PDF: https://arxiv.org/pdf/2608.19842v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

A new reinforcement learning method called Single-rollout Autoregressive Policy Optimization (SAPO) is proposed to address the memory overhead, computation efficiency, and performance issues of existing methods. SAPO uses a shared autoregressive backbone for policy and value functions and introduces a trajectory-level generalized advantage estimator. Experiments show that SAPO outperforms PPO and GRPO in terms of stability and performance while reducing memory cost and runtime.

What the Document Says

• The paper presents Single-rollout Autoregressive Policy Optimization (SAPO), a new method for reinforcement learning.
• SAPO uses a shared autoregressive backbone for policy and value functions.
• It introduces a trajectory-level generalized advantage estimator.
• Experiments show that SAPO outperforms PPO and GRPO in terms of stability and performance.
• SAPO reduces memory cost and runtime compared to PPO.

Key Concepts Mentioned

• Agentic reinforcement learning
• PPO (Proximal Policy Optimization)
• GRPO
• autoregressive backbone
• generalized advantage estimator

Claims Made by the Document

• SAPO addresses memory overhead and computation efficiency issues.
• SAPO outperforms PPO and GRPO in terms of stability and performance.

Evidence Used

• The paper describes the proposed method, SAPO.
• Experiments comparing SAPO, PPO, and GRPO are presented.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks. Despite their success, recent studies revealed three limitations: (1) Lack explicit value generalization and effective temporal credit assignment; (2) Suffer from potential advantage collapse in long-horizon complex tasks; (3) Require a costly trade-off between sampling budget and policy performance. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework in which the policy and value functions share a single autoregressive backbone. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on-policy SARSA objectives. To robustly estimate the contribution of each turn, we further introduce a trajectory-level generalized advantage estimator that combines lambda-returns with batch normalization. Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per-iteration runtime by 33.2% over PPO.