Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

ClawGym II: Exploring Black-Box RL on Agent Harness

Source

arXiv

arxiv.org/abs/2608.16798v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Huatong Song, Fei Bai, Ming Yang, Renyuan Li, Jia Deng, Jujie He, Zhange Zhang, Daixuan Cheng, Yan Xing, Qi Yun, Xuxing Chen, Danyang Li, Feng Chang, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Mingjie Tang, Ji-Rong Wen
  • Venue: arXiv: cs.CL
  • Published: 2026-08-17
  • Collected: 2026-08-18
  • URL: http://arxiv.org/abs/2608.16798v1
  • PDF: https://arxiv.org/pdf/2608.16798v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

Researchers present a new framework for reinforcement learning (RL) using complex agent harnesses. This framework is designed to optimize RL more stably and efficiently for long-horizon tasks. The researchers' method isolates task environments and harnesses, decouples policy optimization from harness execution, and adapts PPO and GRPO algorithms to the new tree structure. The framework also maintains training-inference consistency and introduces mix-harness training. Testing on ClawGym-Bench shows improvements in Pass@1 by 9.98 and 14.81 points for OpenClaw and Claude Code, respectively. The method also performs well on more challenging tasks like JobBench and OfficeQA.

What the Document Says

  • The researchers introduce a new black-box RL framework for optimizing general agents through complex harnesses.
  • They build a sandbox-based execution infrastructure to isolate task environments and harnesses.
  • They decouple policy optimization from harness execution, placing a serving proxy at the model boundary.
  • They capture model calls and organize them into prefix trees for reconstructing multi-turn trajectories.
  • They adapt PPO and GRPO algorithms to optimize over the tree structure.
  • Their method maintains training-inference consistency.
  • They introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses.
  • Testing on ClawGym-Bench shows improvements in Pass@1 for OpenClaw and Claude Code.

Key Concepts Mentioned

  • Agent Harness
  • Reinforcement Learning
  • Black-box Framework
  • Sandbox-based Execution Infrastructure
  • Policy Optimization
  • Heterogeneous Harnesses
  • PPO (Proximal Policy Optimization)
  • GRPO (Generalized Reinforcement Policy Optimization)

Claims Made by the Document

  • The new framework improves performance on long-horizon tasks.
  • Their method enables stable and scalable optimization of general agents through complex harnesses.

Evidence Used

  • Improvements in Pass@1 on ClawGym-Bench by 9.98 and 14.81 points

Methods / Approach

  • Building a sandbox-based execution infrastructure
  • Decoupling policy optimization from harness execution
  • Capturing model calls and organizing into prefix trees
  • Adapting PPO and GRPO algorithms to the new tree structure
  • Maintaining training-inference consistency
  • Introducing mix-harness training

Limitations Stated by the Authors

  • Scaling such training to long-horizon agent tasks introduces fundamental challenges.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The text does not explicitly mention any limitations of the presented method. However, in general, black-box RL might have difficulties with understanding the underlying mechanisms of the harnesses.
  • Also, it's important to note that the performance could vary depending on the specific tasks and harnesses being used.

Open Questions

  • How well does this framework perform on a wider range of long-horizon tasks?
  • What are the potential pitfalls or challenges when applying this method in real-world scenarios?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • This research focuses on improving RL performance on long-horizon tasks using complex harnesses. However, it would be interesting to explore its applicability and performance across various domains and tasks.

Raw Abstract / Excerpt

Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.