Trend

Coding agents can produce faster than teams can verify

Day · 2026-07-02 · Software Intelligence

Coding-agent work today centers on verification under scale. Enterprise telemetry shows pull-request output doubling, while UnderSpecBench and TestEvo-Bench expose two pressure points: agents act under ambiguity, and tests must track real code changes.

Enterprise coding throughput

A large field study gives the clearest production signal in the period. In one mid-sized B2B software company, per-developer merged pull requests reached 2.09 times the pre-mandate baseline by April 2026. The panel covered 802 developers and 196,212 non-bot pull requests across 364 repositories.

The gain came with a visible review cost. AI-authored pull requests reached about 90% by the end of the study window, per-reviewer load roughly doubled, and automated review overtook human review. Merge and revert rates stayed about flat, so the short-horizon quality signals did not collapse in the reported data.

A smaller SAP field study adds a developer-experience view. Developers liked generative AI most for repetitive and structured tasks. The study also reports that mixing in-code suggestions and chat prompts within the same task reduced the benefit, which suggests tool design still affects workload even when output volume rises.

Action boundaries and code-review controls

Safety results are strongest where agents can touch operational state. UnderSpecBench tests DevOps requests with missing action, target, or scope. Across five agent-model setups, safe success ranged only from 15.5% to 36.8%. Among acted runs, 55.8% to 67.8% crossed a boundary through wrong-target or over-scope behavior.

Persistent codebases add another failure mode. Iterative VibeCoding shows attacks split across several pull requests can evade monitors that inspect each diff in isolation. A link-tracker monitor reduced evasion, but the reported evasion rates remained high enough to keep sequence-level review in scope.

One mitigation paper focuses on constraints inside the codebase. In a small Python backdoor test, enforceable contracts plus a 200-line documentation command-line interface raised Gemma 4 e4b reviewer recall to 90.9%. The experiment is small, but the mechanism is concrete: make invariants easy to inspect and hard to bypass.

Executable benchmarks for changing code

TestEvo-Bench targets a practical maintenance task: updating or adding tests after real code changes. The benchmark mines Java Maven repositories, checks builds and tests, and asks agents to write tests that pass on the new revision and fail on the old one. The released snapshot includes 746 test-generation tasks and 509 test-update tasks from 152 open-source projects.

Top reported success sits below full automation. Claude Code with Claude Opus 4.7 and Gemini CLI with Gemini 3.1 Pro both reached 77.5% success on test generation. On test update, the best reported result was 74.6%. Mutation scores on passing outputs were lower than success rates, which matters because a passing test can still miss the changed behavior.

DualView attacks a related repository-scale problem. It gives issue-resolution agents visual and textual graph views of module coupling, calls, class hierarchy, and program dependence. The paper reports up to 388 solved SWE-bench Pro instances and a 46-instance gain over its stated OpenCode plus Kimi K2.5 baseline.

Reliability settings and hidden costs

One repeated-build study tested 90 independent agent sessions on the same real-time retrospective-board application. Frontier models scored near the 42-point ceiling, but first-try reliability depended strongly on reasoning effort. Raising Claude Opus 4.7 from High to xHigh increased perfect first attempts from 28% to 89% and cut corrective prompts sharply, with a median cost increase of 9% to 29%.

Extra tool access did not pay off in this setting. Playwright raised median cost by 42% to 68% on Opus 4.7 without improving functional score or reliability. The added cost came mainly from context re-reading. The practical signal is narrow but useful: when failures are reasoning or environment setup failures, a browser-testing tool may add expense without removing repair work.

NewerRobot policies are being rebuilt around control-loop evidenceOlderRobot learning papers put VLA policies under execution pressure