TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jiale Amber Wang, Kaiyuan Wang, Pengyu Nie
• Venue: arXiv: cs.SE
• Published: 2026-07-02
• Collected: 2026-07-03
• URL: http://arxiv.org/abs/2607.02469v1
• PDF: https://arxiv.org/pdf/2607.02469v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have developed TestEvo-Bench, a live benchmark for testing software tests and code co-evolution. The benchmark includes two tracks: test generation and test update, where an agent should either write new tests or adapt existing ones to match changes in the code. Each task is anchored to real commit history, packaged with environment configuration, and recorded with a timestamp to support execution-grounded metrics. TestEvo-Bench contains 746 test generation tasks and 509 test update tasks curated from 152 open-source Java projects. Four state-of-the-art agents were tested on the benchmark, achieving up to 77.5% success rate in test generation and 74.6% in test updates. However, success rates dropped significantly under limited per-task cost and were lower for more recent tasks.

What the Document Says

• TestEvo-Bench is a live benchmark for evaluating test and code co-evolution.
• The benchmark includes two tracks: test generation and test update.
• Each task in the benchmark is anchored to real commit history, packaged with environment configuration, and recorded with a timestamp.
• TestEvo-Bench contains 746 test generation tasks and 509 test update tasks curated from 152 open-source Java projects.
• Four state-of-the-art agents were tested on the benchmark.
• The agents achieved up to 77.5% success rate in test generation and 74.6% in test updates.

Key Concepts Mentioned

• TestEvo-Bench
• test and code co-evolution
• benchmark
• Java projects
• state-of-the-art agents

Claims Made by the Document

• The benchmark helps evaluate whether a test automation agent understands how a code change should propagate into the test suite.

Evidence Used

• Making it difficult to evaluate existing test generation and update benchmarks.
• Introducing TestEvo-Bench as a solution.

Methods / Approach

• Developing a benchmark of test and code co-evolution tasks mined from software repositories.
• Experimenting with four state-of-the-art agents on the developed benchmark.

Limitations Stated by the Authors

• Success rate is lower on more recent tasks and drops significantly under limited per-task cost.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The text does not specify any potential limitations of the agent or methodology used in this research.
• It's possible that the agents might not perform well on complex or less common co-evolution scenarios.

Open Questions

• Can the agents handle more complex or less common test and code co-evolution scenarios?
• How does the success rate of the agents compare to other methods in similar tasks?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The study focuses on evaluating existing state-of-the-art agents, so it's important to consider other potential approaches or methods when exploring this topic further.

Raw Abstract / Excerpt

Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite. We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.