Recoleta
GitHub
Source note

To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair

Authors
Zhihao Lin, Junhua Zhu, Mingyi Zhou, Xin Wang, Zhensu Sun, Renyu Yang, et al.
Source
arXiv
Published
Collection
Software Intelligence
Program RepairCode AgentsSwe BenchTest ExecutionCost Effectiveness
Open arXivMarkdown source

Summary

The paper finds that letting LLM repair agents run tests adds little repair accuracy on the studied SWE-bench tasks while adding large token, time, and environment costs. Its main claim is that agents should decide when execution is worth paying for instead of running tests by default.

Problem

  • LLM program-repair agents often use a loop of inspecting code, editing, running tests, and revising patches.
  • Running project tests costs tokens, wall-clock time, and per-repository environment setup.
  • Prior SWE-bench results usually mix agent design changes with execution access, so they do not isolate how much test execution itself helps.

Approach

  • The study first analyzes 7,745 public SWE-bench agent traces from SWE-agent, OpenHands, LiveSWEAgent, and Mini-SWE-agent across 12 LLMs.
  • It then runs 3,000 controlled repair attempts on 200 SWE-bench instances: 100 Lite and 100 Verified.
  • The controlled runs use Claude Code with Claude Sonnet 4.5, Codex with GPT-5.2-xhigh, and OpenCode with Qwen2.5-Coder-32B-Instruct.
  • The researchers keep the agent and task fixed while changing execution access: Prohibited, Quota-1, Quota-3, Budget-Guided, and Unrestricted.
  • They measure resolve rate, token use, wall-clock time, execution count, localization accuracy, single-edit ratio, and whether agent-run tests match official SWE-bench evaluation.

Results

  • In 7,745 public traces, agents run tests on average 8.8 times per task, with a range of 2.0 to 18.7 executions per task across agent-model pairs.
  • Late-stage executions have higher pass rates than early executions; one reported case, OpenHands with Claude-3.5-Sonnet, rises from 42% early to 72% late, and the average execution success rate is 57.9%.
  • On commercial agents, the resolve-rate gap between Prohibited and Unrestricted is 1.25 percentage points and is not statistically significant by McNemar's test (p > 0.05).
  • Claude Code resolves 63% without execution and 64% with unrestricted execution, while Prohibited saves 56% of tokens and 48% of wall-clock time.
  • OpenCode with Qwen2.5-Coder-32B resolves 10% in both Prohibited and Unrestricted modes and uses about 3x fewer tokens without execution.
  • For commercial agents, 54-66% of cases finish in a single edit, localization accuracy under Prohibited stays above 95%, and 81-100% of failed cases pass the agent's own validation but fail official SWE-bench evaluation.

Link