Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Peng Xu, Sijia Chen, Junzhuo Li, Xuming Hu
• Venue: arXiv: cs.LG
• Published: 2026-06-24
• Collected: 2026-06-25
• URL: http://arxiv.org/abs/2606.25852v1
• PDF: https://arxiv.org/pdf/2606.25852v1
• DOI: UNKNOWN

Plain-Language Brief

The researchers propose a method called Semantic Consistency Policy Optimization (SCPO) to address a problem in reinforcement learning of LLM agents. The issue is that these agents are trained for long-term, sparse reward tasks using group-based reinforcement learning, but this method can lead to semantic inconsistencies where similar actions receive opposite credit depending on the outcome of the entire trajectory. SCPO aims to resolve this inconsistency by recovering step-level credit from successful siblings within the same rollout group, thereby improving performance on tasks such as ALFWorld and WebShop.

What the Document Says

• Group-based reinforcement learning is used for post-training LLM agents for long-horizon, sparse-reward tasks.
• The method ties a step's credit to its rollout's final outcome, leading to semantic inconsistency.
• The inconsistency sends conflicting gradients to similar actions and wastes partially-correct progress in failed rollouts.
• SCPO is proposed as a value-free reward-shaping method to mitigate this inconsistency.
• SCPO recovers step-level credit from successful siblings within the same rollout group.
• SCPO matches or exceeds strong group-based baselines on ALFWorld and WebShop tasks.
• On ALFWorld, SCPO achieves 93.7+/-4.1 percent success at 1.5B parameters.
• On WebShop, SCPO achieves 74.8+/-2.0 percent success at 1.5B parameters.

Key Concepts Mentioned

• LLM agents
• reinforcement learning
• group-based reinforcement learning
• semantic consistency
• credit assignment
• value-free reward-shaping
• SCPO
• ALFWorld
• WebShop

Claims Made by the Document

• Group-based reinforcement learning ties a step's credit to its rollout's final outcome.
• Semantic inconsistency leads to conflicting gradients for similar actions and wastes partially-correct progress in failed rollouts.
• SCPO mitigates semantic inconsistency by recovering step-level credit from successful siblings within the same rollout group.
• SCPO matches or exceeds strong group-based baselines on ALFWorld and WebShop tasks.

Evidence Used

• Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks.
• Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts.
• Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO)
• Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling.
• On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines
• On ALFWorld, SCPO reaches 93.7+/-4.1 percent success on ALFWorld
• On WebShop, SCPO reaches 74.8+/-2.0 percent on WebShop at 1.5B parameters

Methods / Approach

• Group-based reinforcement learning
• SCPO

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The paper does not provide details about the specific algorithms or implementations of group-based reinforcement learning and SCPO.
• The study only evaluates the performance on ALFWorld and WebShop tasks, and it's unclear how the methods would perform in other domains.

Open Questions

• What are the specific algorithms or implementations of group-based reinforcement learning and SCPO?
• How does the proposed method perform in other domains beyond ALFWorld and WebShop?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper discusses LLM agents but does not specify what LLM stands for. It may refer to Language Learning Models.
• The success rates on ALFWorld and WebShop are given as percentages with error margins.

Raw Abstract / Excerpt

Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.