Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

Source

arXiv

arxiv.org/abs/2607.02469v1

Read the source

Source

Plain-Language Brief

The paper introduces TestEvo-Bench, a new benchmark for testing software that evolves with its code. This benchmark is designed to evaluate how well a test automation agent understands the impact of a code change on the test suite. The benchmark consists of tasks mined from 152 open-source Java projects and includes two tracks: test generation and test update. Each task is tied to a real commit history and comes with environment configuration for execution-grounded metrics. The current snapshot contains 746 test generation and 509 test update tasks. Four state-of-the-art agents were tested on this benchmark, achieving up to 77.5% success rate in test generation and 74.6% in test update, but the success rate drops significantly under limited per-task cost, especially on more recent tasks.

What the Document Says

  • TestEvo-Bench is a benchmark for testing software that evolves with its code.
  • The benchmark consists of two tracks: test generation and test update.
  • Each task is anchored to a real commit history and packaged with environment configuration.
  • The current snapshot contains 746 test generation and 509 test update tasks, curated from 152 open-source Java projects.
  • Four state-of-the-art agents were tested on this benchmark: Claude Code, Gemini CLI, SWE-Agent, Claude Opus 4.7, and Gemini 3.1 Pro.
  • The agents achieved up to 77.5% success rate in test generation and 74.6% in test update.
  • The success rate drops significantly under limited per-task cost, especially on more recent tasks.

Key Concepts Mentioned

  • TestEvo-Bench
  • test automation agent
  • code evolution
  • Java projects
  • state-of-the-art agents

Claims Made by the Document

  • The benchmark helps evaluate understanding of code change impact on test suite.
  • Four state-of-the-art agents were tested with success rates up to 77.5% in test generation and 74.6% in test update.

Evidence Used

  • Isolated existing test generation and update benchmarks do not verify executability or semantic tie to code change.
  • TestEvo-Bench is designed to address these issues.
  • Results show that success rate drops significantly under limited per-task cost, especially on more recent tasks.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite. We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.