EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Zishan Xu, Zhiyuan Yao, Yuxin Chen, Yifu Guo, Zhengxi Lu, Yuquan Lu, Jinyang Huang, Yan Xu, Yasheng Wang, Weinan Zhang, Xingshan Zeng, Weiwen Liu
• Venue: arXiv: cs.AI
• Published: 2026-08-06
• Collected: 2026-08-07
• URL: http://arxiv.org/abs/2608.06197v1
• PDF: https://arxiv.org/pdf/2608.06197v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces EnvACE, a method for training large language model agents for long-term tool use without relying on interactions with real or synthesized environments, which can be costly to construct and verify, or external simulators that are hard to ground. Instead, it uses world rehearsal where the policy alternates between acting and rehearsing. The agent internalizes the relationship between actions and environment responses during training, leading to improved performance on various benchmarks compared to environment-scaling baselines.

What the Document Says

• The paper presents an agentic reinforcement learning method called EnvACE that uses world rehearsal for training large language model agents.
• EnvACE replaces external environment interaction during training with world rehearsal.
• The policy alternates between acting and rehearsing: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response.
• Both roles are jointly optimized end-to-end using task-success rewards.
• The internalized world model supports decision making by yielding an agent world model that directly supports decision making.
• Across several benchmarks (BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench), EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines.
• Controlled studies further show that world rehearsal improves policy learning across model scales.

Key Concepts Mentioned

• reinforcement learning
• world rehearsal
• agentic
• large language model
• tool use
• environment dynamics

Claims Made by the Document

• EnvACE replaces external environment interaction during training.
• EnvACE improves performance on various benchmarks compared to environment-scaling baselines.

Evidence Used

• The paper presents an agentic reinforcement learning method called EnvACE that uses world rehearsal for training large language model agents.
• Across several benchmarks (BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench), EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines.

Methods / Approach

• jointly optimizing end-to-end using task-success rewards

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.