Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

TOM-SWE: User Mental Modeling For Software Engineering Agents

Source

OpenReview

openreview.net/forum

Read the source

Source

Plain-Language Brief

Researchers have developed a new system called ToM-SWE that helps software engineering agents understand and follow user instructions better, even when they are context-dependent or underspecified. The system uses a dual-agent architecture with one agent focused on the user's mental state (ToM agent) and another on software engineering tasks (SWE agent). In two benchmarks and a three-week study with professional developers, ToM-SWE showed improvements in task success rates and user satisfaction compared to existing systems.

What the Document Says

  • The research presents ToM-SWE, a dual-agent architecture that combines a primary software-engineering (SWE) agent with a lightweight theory-of-mind (ToM) partner agent.
  • The ToM agent infers user goals, constraints, and preferences from instructions and interaction history.
  • It maintains a persistent memory of the user and provides user-related suggestions to the SWE agent.
  • Preserving privacy and minimizing context window load are considerations.
  • In two software engineering benchmarks (ambiguous SWE-bench and stateful SWE-bench), ToM-SWE improves task success rates and user satisfaction.
  • On the stateful SWE benchmark, ToM-SWE achieves a higher task success rate of 59.7% compared to 18.1% for OpenHands.
  • In a three-week study with professional developers using ToM-SWE, participants found it better aligned with their intent and useful 86% of the time.

Key Concepts Mentioned

  • software engineering agents
  • user mental modeling
  • dual-agent architecture
  • ToM agent
  • SWE agent
  • user intent
  • interaction history
  • privacy
  • context window load
  • task success rates
  • user satisfaction
  • OpenHands
  • stateful SWE benchmark
  • professional developers

Claims Made by the Document

  • ToM-SWE improves task success rates and user satisfaction.
  • On the stateful SWE benchmark, ToM-SWE achieves a higher task success rate than OpenHands.
  • Professional developers found ToM-SWE better aligned with their intent and useful 86% of the time.

Evidence Used

  • Improved task success rates and user satisfaction in two benchmarks and a three-week study with professional developers

Methods / Approach

  • Using a dual-agent architecture that infers user goals, constraints, and preferences from instructions and interaction history.
  • Preserving privacy and minimizing context window load.

Limitations Stated by the Authors

The paper does not discuss potential limitations of ToM-SWE beyond the need to preserve privacy and minimize context window load.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The system's effectiveness may be limited by factors such as the complexity of user instructions, the specific context of software engineering tasks, and the quality of the interaction history.

Open Questions

  • How does ToM-SWE perform on other types of software engineering benchmarks?
  • What is the impact of ToM-SWE on productivity and efficiency in a real-world setting?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper does not provide information about the specific metrics used to measure task success rates or user satisfaction. It also does not discuss how the study with professional developers was conducted.

Raw Abstract / Excerpt

Recent advances in coding agents have made them capable of planning, editing, running, and testing complex code bases. Despite their growing ability in coding tasks, these systems still struggle to infer and track user intent, especially when instructions are underspecified or context-dependent. To bridge this gap, we introduce ToM-SWE, a dual-agent architecture that pairs a primary software-engineering (SWE) agent with a lightweight theory-of-mind (ToM) partner agent dedicated to modeling the user's mental state. The ToM agent infers user goals, constraints, and preferences from instructions and interaction history, maintains a persistent memory of the user, and provides user-related suggestions to the SWE agent, while preserving privacy and minimizing context window load. In two software engineering benchmarks (ambiguous SWE-bench and stateful SWE-bench), ToM-SWE improves task success rates and user satisfaction. Notably, on the stateful SWE benchmark, a newly introduced evaluation that provides agents with a user simulator along with previous interaction histories, ToM-SWE achieves a substantially higher task success rate of 59.7% compared to 18.1% for OpenHands, a state-of-the-art SWE agent. Furthermore, in a three-week study with professional developers using ToM-SWE in their daily work, participants found it better aligned with their intent and useful 86% of the time, underscoring the value of stateful user modeling for practical coding agents.