TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

Abstract: Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite. We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.

• The paper presents a novel benchmark that mines Java repositories and verifies test-code co-evolution through a robust three-phase pipeline.
• It evaluates state-of-the-art agents, achieving 77.5% success in test generation and 74.6% in test update while exposing harness fragility challenges.
• The benchmark emphasizes the need for dynamic, execution-based assessments to enhance test maintenance, supporting future expansion to more languages and models.

TestEvo-Bench: An Executable, Live Benchmark for Test and Code Co-Evolution

Introduction

TestEvo-Bench introduces a significant advancement in the benchmarking of test-code co-evolution tasks, directly addressing the need for data-driven, execution-grounded evaluation of automated agents that generate or update regression tests in response to software changes. Unlike prior datasets that often rely on static heuristics or metadata, TestEvo-Bench grounds its tasks in real repository history, verifies execution, and tracks precise behavioral relationships between code modifications and associated test edits. This benchmark uniquely enables researchers to assess whether test generation and update agents genuinely reason about the propagation of software behavior changes.

Figure 1: TestEvo-Bench is constructed through a three-phase mining, cleaning, and task construction pipeline.

Benchmark Construction and Design

TestEvo-Bench is constructed via a meticulous three-phase pipeline: repository mining, execution-backed data cleaning, and robust task construction.

• Mining Phase: The benchmark mines open-source Java Maven repositories, filtering for projects with stable build and execution properties. Commit-pairs are selected where both code and test changes co-occur, retaining only high-quality candidates validated via dynamic dependency analysis.
• Execution-Backed Cleaning: Each candidate is rigorously rebuilt and tested across revision pairs. Only those changes that yield executable, semantically meaningful code/test co-evolution pairs are retained. Aggressive filtering discards any tasks lacking a clear behavioral tie between the test and the code change, or where reproducibility is impaired.
• Task Construction: Validated co-evolution instances are formed into benchmark tasks for either test generation (requiring tests that exercise newly introduced software behavior) or test update (demanding updates to failing tests such that they pass with the changed code). Each task is annotated with fine-grained metadata and anchored to real revision timestamps.

  Figure 3: Three-phase mining pipeline used to build TestEvo-Bench: repository-level filtering, commit-level filtering with runtime dependency mapping, and test-level identification of code/test co-evolution triples.

This approach ensures that every benchmark task is strictly executable, fully contextualized, and semantically non-redundant. The datastream is "live": new tasks are regularly harvested, enabling contamination-aware evaluation (i.e., assessments that respect model training cutoffs by filtering out potentially leaked tasks).

Figure 4: TestEvo-Bench task counts across time-segmented subsets. Each bar reports the number of tasks whose new revision falls in the corresponding half-year; the Older subset combines all tasks before 2015.

Experimental Setup and Metrics

TestEvo-Bench evaluates four state-of-the-art agents combining leading code-editing harnesses (Claude Code, Gemini CLI, and SWE-Agent) with modern foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). The benchmark covers two tracks:

• Test Generation: The agent must author new test cases capturing new code behavior. Success requires passing the new revision and failing (or not compiling) on the old revision—a strong test of semantic discrimination.
• Test Update: The agent modifies broken tests so they pass with changed code. Any patch achieving passing and correct compilation is valid, with no requirement to mimic the developer’s text exactly.

Performance is measured using execution-grounded metrics:

• Success Rate: Percentage of tasks where the agent produces a test passing on the new revision (and failing on the old, for generation tasks).
• Coverage: Proportion of changed code lines exercised by the test (collected via JaCoCo).
• Mutation Score: Share of artificial mutants killed by the test (using Universal Mutator), assessing the thoroughness of behavioral testing.
• Failure Modes: Redundant (non-discriminating) tests, execution failures, compilation failures, and harness failures are separately tracked.

Results and Analysis

Test Generation

The best harness-model compositions (Claude Code / Gemini CLI with their respective foundation models) reach 77.5% success rate on test generation. Redundant outputs—tests that pass before and after the change—remain substantial across all agents, reflecting the non-trivial challenge of producing genuinely discriminating oracles. The highest coverage on passed tests closely approaches that of human-authored tests (76.8% vs. 80.8%), with mutation scores suggesting that LLM-generated tests can match or outperform developer-written ones in killing code mutants in passing cases.

Test Update

For test update, top agents achieve a 74.6% success rate. Coverage on passing tasks is again high (79.4%–79.1%), tightly approximating developer reference. Notably, harness failures are a significant limiting factor for some agent configurations, especially the academic SWE-Agent, revealing driver/model interface fragility.

Figure 5: Success across time-segmented subsets for (a) test generation and (b) test update. Subsets with fewer than 20 methods are omitted to suppress small-sample noise.

Temporal and Cost Analysis

Performance declines visibly on recent tasks for test generation, highlighting ongoing risks of training set leakage and dataset shift. Test update appears temporally more stable. When imposing tighter cost budgets per task, success rates drop sharply—most dramatically for Claude Opus 4.7-based agents—which underlines the urgent need for more cost-efficient agentic design for practical, routine deployment.

Figure 6: Empirical CDF when per-task cost is limited. Top row: test generation; bottom row: test update. Vertical dashed lines mark per-task budget caps at \$3 (red), \$1 (green), and \$0.5 (blue); the annotated ``\% kept'' is the fraction of tasks whose default-budget cost stays at or below each cap.

Implications and Future Developments

TestEvo-Bench underscores several important findings:

• Current SOTA agents remain far from saturating these tasks; success rates drop on recent commits and with practical compute constraints.
• Agent reliability is strongly sensitive to harness quality and model-harness interoperation, as evidenced by high harness failure rates outside of industrial agent/model pairings.
• Not all LLM-produced tests offer the intended discriminative power; high rates of redundant tests reveal limitations in test-oracle reasoning.

Practically, the benchmark offers an immediate vehicle for transparent, reproducible assessment of coding agents for regression test generation and maintenance. Theoretically, it highlights the persistent gap between shallow code-diff-based label heuristics and true behavioral change comprehension—emphasizing the value of execution-based, dynamic labels for future learning.

Looking forward, principal directions include: expanding to additional languages and build systems (especially Python and JavaScript), supporting wider model harness coverage (including open-source models), and optimizing the infrastructure for efficient, public-facing execution-based evaluation.

Conclusion

TestEvo-Bench comprehensively raises the bar for the evaluation of LLM-driven test maintenance and generation agents by requiring agents to faithfully reflect code/test co-evolution as it occurs in real repositories. Its executable, dynamically labeled benchmarks expose performance ceiling, model-capture artifacts, and interface fragility in existing systems, and provide an essential basis for continued progress in agentic software engineering. The live, continually growing corpus and contamination-aware subsets ensure it will remain relevant as models and agents evolve. TestEvo-Bench is publicly available with infrastructure and leaderboard at https://www.testevo-bench.com.