Recoleta
GitHub
Source note

Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution

Authors
Ye Shang, Quanjun Zhang, Haichuan Hu, Chunrong Fang, Liang Xiao, Zhenyu Chen
Source
arXiv
Published
Collection
Software Intelligence
Test EvolutionCoding AgentsSoftware TestingCode IntelligenceBenchmarking
Open arXivMarkdown source

Summary

TEBench is a project-level benchmark for coding agents that must update tests after a production-code commit. It matters because current agents find only about half of affected tests, and stale or missing tests can leave changed behavior unchecked.

Problem

  • Existing test-evolution benchmarks give the affected test method as input, so they skip the search across a repository.
  • Real commits can break tests, leave passing tests semantically stale, or add behavior with no test at all.
  • Poor test evolution lets suites pass while no longer checking the behavior changed by the commit.

Approach

  • The task input is a repository and a production-code-changing commit; the output is a test patch.
  • TEBench labels each instance as Test-Breaking, Test-Stale, Test-Missing, or a combination of these types.
  • The dataset comes from Defects4J Java projects after static filtering, execution checks, and quality filters.
  • Evaluation separates test identification from patch quality, using precision, recall, and F1 for affected tests and executability, coverage overlap, and token-level modification similarity for updates.
  • The paper evaluates seven agent configurations across Claude Code, Codex CLI, and OpenCode, plus a heuristic baseline.

Results

  • TEBench contains 314 task instances from 10 Java projects, drawn from 67,670 starting commits and 14 Maven Defects4J projects.
  • The final dataset covers 172 Test-Breaking tasks, 207 Test-Stale tasks, and 199 Test-Missing tasks; labels can overlap.
  • Seven agent configurations reach identification F1 scores from 45.7% to 49.4%, with less than 4 percentage points between them.
  • Test-Stale is hardest, with average F1 around 36%, because agents mainly follow execution failures and miss passing tests that need semantic updates.
  • Exhaustive structural dependency analysis reaches only 66% recall, leaving about one-third of affected tests outside direct dependency tracing.
  • The paper reports that agent patches often compile and run, but their edits differ a lot from developer-written ground truth.

Link