ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Zhongyuan Peng, Dan Huang, Chuyu Zhang, Caijun Xu, Changyi Xiao, Shibo Hong, David Lo, Lin Qiu, Xuezhi Cao, Jiyuan He, Yixin Cao
• Venue: arXiv: cs.AI
• Published: 2026-07-23
• Collected: 2026-07-24
• URL: http://arxiv.org/abs/2607.21217v1
• PDF: https://arxiv.org/pdf/2607.21217v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces ICAE-Bench, a benchmark for evaluating coding agents in interactive project building settings. Instead of being limited to completing code under fully specified instructions, these agents are expected to transform incomplete product intent into working software. The benchmark uses a realistic and reproducible user simulation, derived from precise open-source repositories with executable behavior, and employs standardized tests and multi-dimensional diagnostics for evaluation.

What the Document Says

• The paper presents ICAE-Bench, a benchmark for evaluating coding agents in interactive project building settings.
• Agents are expected to transform incomplete product intent into working software.
• The benchmark uses realistic and reproducible user simulation derived from precise open-source repositories with executable behavior.
• ICAE-Bench employs standardized black-box tests and multi-dimensional diagnostics for evaluation.

Key Concepts Mentioned

• Coding agents
• Interactive project building
• Product requirements
• User Agent
• Open-source repositories
• Benchmark
• Evaluation
• Black-box tests
• Multi-dimensional diagnostics

Claims Made by the Document

• Existing benchmarks do not fully catch up with the shift in what coding agents are expected to do.
• ICAE-Bench is designed to evaluate coding agents under interactive project-building settings.

Evidence Used

• The authors describe the emergence of vibe-coding workflows and the expectation for agents to transform incomplete product intent into working software.
• The paper presents the design and purpose of ICAE-Bench as a benchmark for evaluating coding agents.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The recent emergence of vibe-coding workflows is changing what coding agents are expected to do. Instead of merely completing code under fully specified instructions, agents are increasingly expected to transform incomplete product intent into working software by combining various abilities including planning, requirement clarification, tool use, debugging, and repository-level construction. Yet existing benchmarks have not fully caught up with this shift, evaluating agents on static, fully specified tasks. In this paper, we introduce ICAE-Bench, a benchmark for evaluating coding agents under interactive project-building settings. The basic idea is to start from a fuzzy product requirement, simulating the dynamic paradigm with an automated User Agent. To make this setting both realistic and evaluable, ICAE-Bench introduces three key designs. First, to avoid the ambiguity of unconstrained fuzzy requirements, each task derives ambiguity from a precise real open-source repository with executable behavior. Second, to ensure high-quality and reproducible user simulation, ICAE-Bench grounds interaction through User Agent Data, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation artifacts. Third, to evaluate open-ended repositories fairly, ICAE-Bench uses standardized black-box tests together with multi-dimensional diagnostics, including functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality.