SPADE: Self-Play in Adaptive Synthetic Executable Environments

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, Natasha Jaques
• Venue: arXiv: cs.CL
• Published: 2026-08-19
• Collected: 2026-08-20
• URL: http://arxiv.org/abs/2608.19197v1
• PDF: https://arxiv.org/pdf/2608.19197v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers have developed a new framework called SPADE (Self-Play in Adaptive Synthetic Executable Environments). This framework allows language models to generate and improve their own training environments as they learn, making the goal distribution dynamic. The model plays two roles: an Environment Designer that creates executable code for training environments, and a Reasoning Agent that learns to act in these environments. Through experimentation, the authors found that grounding the Environment Designer on documents from a large pretraining corpus and giving it an accumulated environment memory are crucial components for success. Scaling up to 30B-parameter models, SPADE outperforms a fixed-environment baseline across multiple benchmarks by an average of +5.3, and improves tool-use and game settings as well.

What the Document Says

• SPADE is a self-play Reinforcement Learning (RL) framework that allows a single Language Model (LLM) to play two roles: an Environment Designer and a Reasoning Agent.
• The Environment Designer creates complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface.
• The Reasoning Agent learns to act in these environments.
• Each environment is a stateful, multi-turn environment (state transitions, reward functions, and verification code).
• The regret of the Reasoning Agent is estimated using the gap between its reward with and without privileged hints.
• Through extensive experimentation, several components were found to be critical for success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory.
• SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks.
• It also lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale.

Key Concepts Mentioned

• SPADE
• Self-Play RL framework
• Language Model (LLM)
• Environment Designer
• Reasoning Agent
• Regret estimation
• Privileged hints

Claims Made by the Document

• SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks.
• Scaling to 30B-parameter models, SPADE outperforms a fixed-environment baseline across multiple benchmarks.

Evidence Used

• Improvement by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks.
• The margin over the strongest baseline grows with model scale.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.