Coding Agent Verification Controls

Teams accepting pull requests from coding agents can add a CI or review bot that classifies new and modified test files by oracle strength before review. The first useful version can flag tests with no assertions, non-null checks, boolean-only checks, mock-only checks, and snapshot-only checks, then require a reviewer note or stronger assertion for risky changes.

The case is concrete because agent tests often create a verification illusion. All Smoke, No Alarm studied 86,156 test-file patches from 33,596 agent-authored PRs and found that 80.2% had weak or no explicit oracle signals. The same study found that stronger multi-signal oracles were associated with higher merge likelihood after controlling for agent, PR size, repository stars, task type, and language.

A cheap pilot is to run the classifier only on agent-labeled PRs for two weeks and report three numbers: share of changed test files with no behavior check, share with value or error assertions, and reviewer override rate. The tool does not need to block every weak test. It gives reviewers a targeted warning when a patch adds test files that execute code without checking expected behavior.

Code-repair agents should capture a small evidence bundle for each failing run: failing test name, compiler or runtime error, relevant input and expected output, executed statements in the suspected function, in-scope variable values, branch outcomes, and a trace diff after each attempted patch. The agent can then revise a named repair hypothesis after each failed candidate patch.

PracRepair shows the fuller version of this workflow for Java APR. It combines static context from a Code Property Graph with failing-test traces, variable values, branch outcomes, validation diagnostics, and trace-level behavior changes. With GPT-4o, it reports 162 correct fixes on Defects4J V1.2 and 171 on V2.0, plus 93 unique correct fixes compared with ReInFix. A simpler code-correction study supports the same operational pattern: generate code, run it, return compiler errors or failed test details, and ask the model to revise.

The adoption path can start inside an existing test runner. Store the evidence bundle as a JSON artifact beside each failed agent attempt, cap refinement at a small number of rounds, and compare fixes with and without trace fields on recurring bug classes. Syntax and runtime failures are the easiest early targets; logic and algorithm errors need richer examples and stronger tests.

Internal coding-agent evaluations should publish a run card for each score with the model, harness, tool set, environment image, task source, verifier, prompt or skill pack version, retry policy, and cost. The same task set should also include a small ablation table: fixed model with different harnesses, fixed harness with different models, and skill enabled versus disabled when skills encode project rules.

The need is visible in benchmark results. One position paper reports that Terminal-Bench success for fixed Claude Opus 4.6 ranges from 79.8% ± 1.6 with ForgeCode to 58.0% ± 2.9 with Claude Code, a 21.8-point spread tied to harness choice. It also cites SWE-Bench+ leakage and insufficient-test findings, plus cases where resolved SWE-Bench-style patches fail developer-written tests or diverge from gold-patch runtime behavior.

The skill-evaluation work gives a practical measurement pattern for teams that encode repository rules, API patterns, or workflow preferences as agent skills. It generated tasks and hidden rubrics from each skill, then ran agents with and without the skill. Across about 500 skills, 1,000 tasks, 19 agent-model configurations, and 38,000 valid trajectories, relevant skills added 5.5 to 22 points depending on the model. A team can copy the paired-run design for its own skills before rolling them into default agent workstations.