Recoleta
GitHub
Source note

JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java

Authors
Khashayar Etemadi, Zhendong Su
Source
arXiv
Published
Collection
Software Intelligence
Java PerformanceSoftware BenchmarkingAutomated Program RepairCode AgentsTest Generation
Open arXivMarkdown source

Summary

JETO-Bench is a reproducible Java benchmark for execution time improvement patches, built with the JETO-Mine collection and evaluation tool. It targets automated repair and coding-agent evaluation for performance fixes rather than functional bugs.

Problem

  • Existing execution-time patch benchmarks mainly cover Python, C++, or .NET, while Java has fewer reproducible datasets.
  • Java timing is hard to measure because JIT compilation, garbage collection, class loading, and JVM warm-up can distort execution-time comparisons.
  • Fixed benchmark datasets cannot collect new real-world patches with user-chosen filters, build settings, or statistical thresholds.

Approach

  • JETO-Mine crawls GitHub Java repositories and filters commits or PRs by project activity, Java version, Maven Wrapper use, changed files, and issue links.
  • An LLM-based classifier checks whether the linked issue is mainly about execution-time improvement.
  • The dynamic phase builds Docker images for original and patched versions, runs a warm-up round, then runs tests repeatedly under the same JVM setup.
  • It uses paired one-sided binomial tests to decide whether a patched version is faster under user-set thresholds, with defaults of 30 timed runs, 5% minimum improvement, and p-value 10%.
  • The evaluation harness applies generated patches or tests inside the Docker environment, runs project tests, extracts timings, and reports build, test, and speed results.

Results

  • JETO-Mine scanned 3,686 repositories and 1,769,958 commits across 11 years of open-source Java history, up to 2025-11-28.
  • JETO-Bench contains 660 identified execution time improvement patches and 91 manually verified executable ETIPs from 174 Java repositories.
  • The repository pool had at least 20 stars by default; observed stars ranged from 20 to 93,448, with a median of 53.
  • Commit counts per considered repository ranged from 1 to 55,824, with a median of 58.
  • OpenHands fixed 14.3% of the manually verified executable tasks, or 13 out of 91.
  • The paper reports that open-source Java projects often lack tests that show execution-time improvements, which limits automatic validation and points to a test-generation gap.

Link