Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests

Source

arXiv

arxiv.org/abs/2607.00990v1

Read the source

Source

Plain-Language Brief

A study was conducted to improve software engineering agents that generate patches for software issues. The researchers found that using bug reproduction tests (BRTs) directly to guide patch generation can be ineffective or misleading. To address this, they propose a new agent named SWE-Doctor that generates and executes multi-faceted BRTs, creates diagnosis records, and uses the diagnoses to guide patch generation, achieving better results compared to existing agents.

What the Document Says

  • There are software engineering agents that generate patches based on issue reports and code repositories. Bug reproduction tests (BRTs) are useful for validating these patches but it is unclear if they can help with the patch generation process.
  • The researchers found that using advanced BRT generators to guide patch generation is not beneficial due to misleading fail-to-fail BRTs and limited or negative gains from even fail-to-pass BRTs.
  • The authors propose SWE-Doctor, an agent that uses runtime diagnoses derived from multi-faceted BRT executions to guide patch generation, reducing partial patches.
  • SWE-Doctor was evaluated on Python bug-fixing issues from the SWE-bench Verified and SWE-bench Pro across five LLM backends.
  • SWE-Doctor outperformed existing agents in all evaluated combinations and achieved average resolution rates of 75.7% on SWE-bench Verified and 59.4% on SWE-bench Pro.

Key Concepts Mentioned

  • software engineering agents
  • patch generation
  • bug reproduction tests (BRTs)
  • runtime diagnosis
  • SWE-Doctor

Claims Made by the Document

  • Using advanced BRT generators to guide patch generation is not beneficial.
  • SWE-Doctor outperforms existing agents.
  • SWE-Doctor reduces partial patches.

Evidence Used

  • directly using advanced BRT generators to guide patch generation is not beneficial: fail-to-fail BRTs can mislead agents, while even fail-to-pass BRTs bring limited or negative gains.
  • SWE-Doctor was evaluated and consistently outperformed existing agents across all 10 LLM-benchmark combinations.

Methods / Approach

  • preliminary study
  • proposing a new software issue resolution agent named SWE-Doctor
  • generating multi-faceted BRTs
  • executing and debugging these BRTs to construct runtime-grounded diagnosis records
  • using the diagnoses together with localization information inferred during BRT generation to guide patch generation

Limitations Stated by the Authors

  • Motivated by their analysis, they propose SWE-Doctor but do not provide detailed insights on why fail-to-pass BRTs may cover only one manifestation of the reported issue or why fail-to-fail BRTs are unreliable as direct patch-generation targets.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The research does not discuss potential limitations that might arise during the implementation or deployment of SWE-Doctor in real-world scenarios.

Open Questions

  • What are the specific reasons why fail-to-pass BRTs may cover only one manifestation of the reported issue?
  • Why are fail-to-fail BRTs unreliable as direct patch-generation targets?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The researchers hint at the reasons behind their findings but do not provide detailed explanations. Further research or discussions with the authors may help clarify these aspects.

Raw Abstract / Excerpt

Large language model (LLM)-based software engineering agents are increasingly developed to resolve software issues by generating patches from issue reports and code repositories. Bug reproduction tests (BRTs) are an important building block for such agents and have been shown useful for patch validation. However, it remains unclear whether BRTs can also help the more central stage of patch generation. We first conduct a preliminary study and find that directly using advanced BRT generators to guide patch generation is not beneficial: fail-to-fail BRTs can mislead agents, while even fail-to-pass BRTs bring limited or negative gains. Our analysis reveals two reasons: fail-to-pass BRTs may cover only one manifestation of the reported issue, leading to partial patches, whereas fail-to-fail BRTs are unreliable as direct patch-generation targets. Motivated by these insights, we propose SWE-Doctor, a software issue resolution agent that guides patch generation with runtime diagnoses derived from multi-faceted BRT executions. SWE-Doctor first generates multi-faceted BRTs for different behavioral requirements stated in the issue, then executes and debugs these BRTs to construct runtime-grounded diagnosis records, and finally uses the diagnoses together with localization information inferred during BRT generation to guide patch generation and reduce partial patches. We evaluate SWE-Doctor on Python bug-fixing issues from the widely adopted SWE-bench Verified and SWE-bench Pro across five LLM backends. SWE-Doctor consistently outperforms existing agents across all 10 LLM-benchmark combinations, achieving average resolution rates of 75.7% on SWE-bench Verified and 59.4% on SWE-bench Pro. In particular, on the more challenging SWE-bench Pro, SWE-Doctor improves the average resolution rate by 8.0-8.9 percentage points over the baseline agents.