AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, Chuanhao Li, Xiangcheng Cao, Yihao Liu, Fanrui Zhang, Li Jin, Kaipeng Zhang
• Venue: arXiv: cs.AI
• Published: 2026-07-02
• Collected: 2026-07-03
• URL: http://arxiv.org/abs/2607.02255v1
• PDF: https://arxiv.org/pdf/2607.02255v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces a new contract for long-horizon LLM agents that restricts the amount of past information available during decision-making in the game Slay the Spire 2. The new contract allows for easier isolation of memory components and enables ablation of any single layer. A study shows that the no-store baseline wins 3 out of 10 games, while adding a strategic skill layer increases the win rate to 6 out of 10.

What the Document Says

• The paper presents an alternative bounded contract for long-horizon LLM agents.
• The contract restricts past information available during decision-making by assembling every decision from a fresh user message.
• The contract is instantiated in Slay the Spire 2, a stochastic deck-building game.
• A public online benchmark shows no wins at the lowest difficulty across five configurations of LLMs, while human win rate is reported at 16% at the same difficulty.
• A fixed-A0 ablation study is conducted with strategic skills enabled and shows a difference in win rates: no-store baseline wins 3 out of 10 games and adding the skill layer increases the win rate to 6 out of 10.
• The comparison between baselines is directional rather than statistically decisive.

Key Concepts Mentioned

• long-horizon LLM agents
• bounded contract
• Slay the Spire 2
• stochastic deck-building game
• memory layers
• strategic skills

Claims Made by the Document

• The new contract makes prior context easy to access and isolates the effect of any single memory component.
• A fixed-A0 ablation study shows a difference in win rates when triggered strategic skills are enabled.

Evidence Used

• The paper describes the contract and its design.
• The study results are presented.

Methods / Approach

• Fixed-A0 ablation study is conducted with strategic skills enabled.

Limitations Stated by the Authors

• The comparison between baselines is directional rather than statistically decisive.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The paper does not state any limitations identified by the agent.

Open Questions

• How do different memory layers impact long-horizon LLM-agent decisions in other games or tasks?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The study is directional rather than statistically decisive, which means that while there is a difference observed between the baselines, it may not be statistically significant.

Raw Abstract / Excerpt

Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see. The simplest contract appends past observations, tool calls, and reflections to every prompt, which makes prior context easy to access but also turns it into a jumbled mixture in which the effect of any single memory component is hard to isolate. We introduce and instrument an alternative bounded contract: every decision is made from a fresh user message assembled by typed retrieval, with no raw cross-decision transcript appended. The prompt thus stays bounded across runs of any length, and any single layer can be ablated in isolation. We instantiate the contract in Slay the Spire 2, a closed-rule stochastic deck-building game whose runs require hundreds of tactical and strategic decisions. A public online benchmark of frontier LLMs on the same game reports zero wins at the lowest difficulty across five configurations, and the developer-reported human win rate at the same difficulty is 16%; the task is hard but not saturated. Within our harness, a fixed-A0 ablation shows the largest observed difference when triggered strategic skills are enabled: the no-store baseline wins 3/10 games and adding the skill layer 6/10. At this sample size the comparison is directional rather than statistically decisive (Fisher exact p\approx0.37); a cross-backbone probe and public accumulating-context baselines are reported as operational comparisons rather than controlled tests of the contract variable itself. We release a reproducible testbed: 298 completed trajectories with condition tags, frozen memory/skill snapshots, prompt records, and analysis scripts -- an agent design and a validated, reusable methodology for studying how explicit memory layers shape long-horizon LLM-agent decisions.