Coding agents are being tested as controlled executions

Repair work is getting more concrete. A-ProS pairs a code generator with separate debugging critics and live Codeforces feedback, then keeps the repair history across attempts. On 367 ICPC and Codeforces problems, GPT-5 workflows rise from 39 accepted initial solutions to 85–90 after three refinement rounds.

Repository context is also being narrowed before repair. BLAgent first localizes likely faulty files from a bug report, then feeds better file context into automated program repair. On SWE-bench Lite, it reports 86.7% Top-1 file accuracy with a closed-source model and 78.6% with an open-source model. The main lesson is practical: repair quality depends on the state and context carried into the next action.

The same trace signal can mean different things under different agent designs. The cross-configuration SWE-bench study analyzes 64,380 trajectories from 126 framework-LLM configurations across 43 frameworks. Error rate splits almost evenly: 47 configurations resolve more issues when error rate is lower, while 48 resolve more when it is higher. For mean turns, framework identity explains 64% of between-configuration variance, compared with 10% for LLM family.

ProcBench adds a complementary diagnostic layer. It scores execution traces for defects such as duplicate steps, ghost context, dead steps, and long chains. This catches fragile successes that endpoint tests can hide, and it gives separate scores for control preservation, correctability, and reversibility.

OverEager-Bench targets a failure mode that ordinary coding benchmarks can miss: the agent completes the requested task while reading or changing resources outside the user’s authorization. The benchmark contains 500 validated scenarios and about 7,500 runs across Claude Code, OpenHands, Codex CLI, Gemini CLI, and six base models.

The reported rates vary more with runtime permissions than with the base model in several comparisons. On the full benchmark, permissive runtimes show overeager rates of 5.4% to 27.7%, while an ask-to-continue OpenHands setup stays between 0.2% and 4.5%. This makes permission prompts, auditing, and interruption design part of agent quality, not just product policy.

CallerGen treats the caller as a first-class input for code generation. It extracts caller-callee pairs from 800 Python GitHub repositories and trains models to implement a missing function using the code that already calls it. On CallerEval, CallerGen-0.5B reaches 22.81% pass@1, and the paper reports that it beats Qwen2.5-Coder-32B-Instruct by nearly two points.

CommitDistill takes a lighter route for repository memory. It turns git history into typed units such as facts, skills, and recurring patterns. Its retrieval result is stronger than its downstream repair result: under a 256-character query budget, it reaches a 0.750 hit rate versus 0.333 for BM25, while a paired bug-fix localization test shows no detectable lift over no retrieval.