Coding agents need failure-tested guidance and gated execution

Repository context is becoming an object to test, edit, and reuse. Probe-and-Refine starts with generated repository guidance, runs synthetic bug-fix probes, and folds the failures back into a compact guidance file. On 500 SWE-bench Verified instances, it reports a 33.0% mean resolve rate, compared with 28.3% for static repository guidance and 25.5% with no context. The gain mainly comes from more evaluable patches, which suggests that better guidance helps the agent find the right area of the codebase.

Phoenix applies a similar engineering instinct to GitHub issue resolution. It splits work across planner, reproducer, coder, tester, failure analyst, and pull-request agents. Its tester compares a clean baseline run with the patched run, then opens a pull request only when the change adds no new failing tests. On a curated 24-task SWE-bench Lite slice, Phoenix reports 18 oracle-resolved tasks and no PASS_TO_PASS regressions on successful runs.

Tool permissions are now a measurable agent failure mode. ToolPrivBench gives each task lower-privilege and higher-privilege tools that can all finish the job, so unnecessary high-privilege choice is counted directly. Across 11 models, six exceed 30% over-privileged tool use. Qwen3-8B reaches 64.9%, and LLaMA-3.1-8B reaches 55.9%. General safety tuning transfers poorly in the reported examples, which points to privilege choice as its own target for training and evaluation.

Production-facing agent writeups make the same boundary concrete. Vercel’s Eve packages durable sessions, sandboxes, approvals, channels, traces, and evals into an open-source TypeScript runtime. A separate production brief argues that auth, payments, secrets, and untrusted input need specs, tests, threat modeling, review, audit trails, and named human owners before merge. The common requirement is simple: agents may act quickly, but sensitive changes need explicit gates.

Python-only code scores look too narrow for current claims about coding ability. Multi-LCB extends LiveCodeBench to 12 languages while keeping release-date filtering and hidden tests. It reports that GPT-OSS-120B Medium has the best average in the excerpt at 67.8 Pass@1, while Qwen3-235B-A22B-Thinking-2507 scores higher in Python and C++ but falls in Rust, Ruby, and Go. The result makes cross-language variance visible instead of hiding it behind a single Python number.

JAMER adds another pressure test: project-level game code on Godot. Its pipeline filters more than 240,000 candidate repositories down to 8,133 behavior-valid projects, then evaluates compile success, structural completeness, and runtime behavior. On code completion, runtime pass rates fall from 80.4% on small projects to 5.7% on large ones. Agent runs improve compilation and runtime pass rates in some settings, but the paper reports no gain in runtime behavioral quality.

Reliability work is testing whether cheap agent diversity can mask defects. The N-version programming study generated 69 implementations across five agent harnesses, 23 model configurations, and three languages. After acceptance screening, 48 versions ran on one million randomized inputs. The independence assumption failed: 429 coincident-failure cases appeared where random independence predicted 115.36. Majority voting still helped, cutting mean failures across three-version units from 387.44 for single versions to 130.99.

AutoPass applies the evidence-loop pattern to compiler tuning. It uses LLVM intermediate representation, optimization remarks, runtime measurements, and hardware counters to revise pass pipelines. Across 64 workloads, the three-iteration version reports the best result in 9 of 10 platform-suite settings under the paper’s budget, with geometric-mean speedups over LLVM -O3 of 1.043× on x86-64 and 1.117× on ARM64.