Executable Feedback for Coding-Agent Workflows

Maintainers using repair agents should add a reproduction stage before patch generation. The stage would localize the suspected bug, retrieve repository context, generate a test, and execute it on the affected revision. ReProAgent reproduced 58.43% of SWT-bench-lite issues and 70.30% of SWT-bench-verified issues at an average cost of $0.14 per case; its tests also improved downstream repair performance.

A practical pilot can use 50 closed issues with known fixes. Accept a generated test only when it fails on the pre-fix commit and passes after the recorded fix, then compare patch success, maintainer review time, and false reproductions with the existing repair-agent workflow. Teams that clear this check can require an executable reproduction artifact in the issue before an autonomous repair run begins.

Teams operating coding agents should version the executable harness separately from the model and test proposed harness edits in a shadow runner. Each trial can replay recent execution traces, vary context construction, tool-use rules, verification steps, or recovery logic, and promote a candidate only after it passes held-out repository tasks and cost limits.

TTHE showed the size of this opportunity with frozen model weights: on DeepSeek-V4-Flash, SWE-bench Verified rose from 20.0% to 35.0%. Its documented selection regret and judge errors also show why deployment needs held-out checks. Long-Horizon-Terminal-Bench adds a useful evaluation design: grade executable subtasks throughout the run and record timeouts, since 79% of unresolved runs timed out. A low-cost trial would compare two harness versions on the same model, task set, token budget, and wall-clock limit, tracking final completion, checkpoint progress, regressions, and cost.

Java teams using coding agents for unit-test generation can add a supervisor that reads current line coverage, branch coverage, missed complexity, and token cost after each iteration. The supervisor chooses whether the agent should continue normally, receive program-analysis output for uncovered paths, or stop. This targets premature completion on easy paths and limits repeated calls that add no coverage.

Scate used this pattern with a contextual bandit and an MCP program-analysis tool. Against unsupervised agent baselines, it reported 32.3% higher line coverage and 30.9% higher branch coverage with Gemini CLI, plus gains of 6.0% and 5.9% with Claude Code. A repository pilot should hold the model and token budget constant, then compare branch coverage, mutation score, generated-test maintenance failures, and cost per accepted test. The paper does not report absolute final coverage or statistical significance, so those checks should precede wider adoption.