UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Temitayo Olamilekan Ogunsusi, Lijun Qian, Xishuang Dong
• Venue: arXiv: cs.AI
• Published: 2026-07-02
• Collected: 2026-07-03
• URL: http://arxiv.org/abs/2607.02186v1
• PDF: https://arxiv.org/pdf/2607.02186v1
• DOI: UNKNOWN

Plain-Language Brief

The researchers proposed UA-ChatDev, an uncertainty-aware multi-agent software development framework to improve the reliability of autonomous multi-agent software development. The framework uses a lightweight uncertainty estimation mechanism and phase-aware threshold calibration for retrieval-based verification when the level of uncertainty exceeds acceptable levels.

What the Document Says

• Software development involves collaboration among agents with diverse roles.
• Large language models enable autonomous multi-agent software development frameworks.
• Existing approaches assume equal reliability of intermediate agent outputs, leading to hallucination propagation and impact on final software quality.
• UA-ChatDev is an uncertainty-aware multi-agent software development framework.
• UA-ChatDev integrates uncertainty quantification into agent interactions.
• It uses a lightweight uncertainty estimation mechanism based on token-level log probabilities.
• Phase-aware threshold calibration is employed for retrieval-based verification.
• UA-ChatDev outperforms existing single-agent and multi-agent software development frameworks on completeness, executability, consistency, and overall quality metrics.
• Ablation studies and communication analyses verify that uncertainty-aware interactions enhance code execution reliability.

Key Concepts Mentioned

• software development
• multi-agent collaboration
• autonomous software development
• uncertainties
• uncertainty quantification
• hallucination propagation
• UA-ChatDev
• token-level log probabilities
• phase-aware threshold calibration

Claims Made by the Document

• Existing approaches are vulnerable to hallucination propagation.
• UA-ChatDev addresses this challenge by integrating uncertainty quantification into agent interactions.
• UA-ChatDev outperforms existing single-agent and multi-agent software development frameworks.

Evidence Used

• The text describes the problem of hallucination propagation in existing approaches and presents UA-ChatDev as a solution that addresses this issue.
• The experiments on the SRDD benchmark demonstrate UA-ChatDev's superior performance compared to other frameworks.

Methods / Approach

• UA-ChatDev uses a lightweight uncertainty estimation mechanism based on token-level log probabilities
• Phase-aware threshold calibration is used for retrieval-based verification

Limitations Stated by the Authors

The paper does not provide details about the specific software development tasks, environments, or datasets used in the experiments.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The agent's understanding of the SRDD benchmark and its evaluation metrics is limited by the information provided in the text.

Open Questions

• What specific software development tasks were evaluated in the experiments?
• What environment and datasets were used for the experiments?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text does not provide explicit details about the experimental setup, which raises questions regarding the nature of the software development tasks, environments, or datasets used.

Raw Abstract / Excerpt

Software development is a complex task that demands cooperation among agents with diverse roles. Large language models (LLMs) have enabled autonomous multi-agent software development frameworks that leverage role-based collaboration to automate requirements analysis, coding, testing, and refinement. However, existing approaches typically assume that intermediate agent outputs are equally reliable, leaving them vulnerable to hallucination propagation, where incorrect decisions generated in early development phases are transferred to downstream agents and negatively impact final software quality. To address this challenge, we propose UA-ChatDev, an uncertainty-aware multi-agent software development framework that integrates uncertainty quantification into agent interactions. It introduces a lightweight uncertainty estimation mechanism based on token-level log probabilities to assess the confidence of agent responses and employs phase-aware threshold calibration to selectively trigger retrieval-based verification when uncertainty exceeds acceptable levels. Extensive experiments on the SRDD benchmark demonstrate that UA-ChatDev consistently outperforms existing single-agent and multi-agent software development frameworks across completeness, executability, consistency, and overall quality metrics. Further ablation studies and communication analyses verify that uncertainty-aware interactions enhance code execution reliability.