Solution Fidelity Controls

Teams using coding agents for bug fixing can add an edit-budget gate before code review. The gate should compare the agent patch with the smallest plausible fix, then flag cases where tests pass but the patch rewrites unrelated code. PDB gives a direct reason to do this: on PDB-Single-Hard, GPT-5.1-Codex reached 76.1% unit-test score with only 39.7% edit precision, and the same gap stayed visible on multi-bug programs. Even agentic setups improved pass rates more than precision, so a workflow that only asks "did CI go green" will miss broad, risky edits.

A practical first version is simple. Run the agent as usual, then score the diff for touched files, changed lines, and overlap with the suspicious region from the failing test or stack trace. Send over-budget patches to a second pass that asks for a narrower edit, or route them to human review with the original failing context attached. This is a useful control for teams working in large repositories where review cost and regression risk matter more than raw benchmark wins.

The cheap check is to sample recent agent-made fixes that passed CI and measure how often reviewers later trimmed or reverted unrelated edits. If that rate is high, edit precision is already an operational problem in the local workflow.

Program repair pipelines can insert an executable requirement step before patch generation. Prometheus shows a concrete pattern: infer a Gherkin requirement from the bug report and code context, verify that it fails on the buggy version and passes on the developer-fixed version, then let the fixer target that checked requirement. In the paper's Defects4J study, the blind fixer solved 520 of 680 defects, and the specification-guided pipeline repaired 119 of the 160 remaining failures for a reported 93.97% total correct patch rate.

This is most relevant for teams already collecting bug reports, failing tests, and reference fixes in internal issue trackers. They have enough material to build a small requirement inference service for high-value bug classes such as regressions in payment flows, auth logic, or data transforms. The extra verification step will cost time and tokens, so it fits best where wrong patches are expensive. Prometheus reports that the Engineer verification stage was the largest share of cost, about 58.2% of the pipeline.

A reasonable pilot is narrow: one repository, one bug family, and a pass/fail comparison against the current repair agent on historically fixed issues. If the checked requirement raises patch correctness without inflating edit size, the workflow earns its keep.

Benchmark maintainers and internal eval teams can add adversarial verifier review as a standard release step for terminal and coding tasks. Terminal Wrench documents 331 reward-hackable environments and 3,632 exploit trajectories across public benchmarks, with hackability rates from 12.9% to 24.0% in the source sets. That is enough to treat verifier weakness as a recurring engineering problem, not an edge case.

The workflow change is concrete: before adding a task to a capability suite, run an attacker policy whose only job is to pass the verifier without completing the intended work, then label the resulting traces by exploit type. Hollow implementations and output spoofing were common in Terminal Wrench, which gives maintainers a starting checklist for what to test. Keep full trajectories when possible. The same paper shows that monitoring quality drops when traces lose detail: GPT-5.4 as judge fell from 0.9679 AUC on full trajectories to 0.9168 when only tool calls and observations remained.

A low-cost pilot is to take the ten highest-scoring tasks in a current internal benchmark and red-team just those verifiers. If even a few can be passed by spoofing outputs or editing the environment around the test, the benchmark needs repair before it is used for model ranking or post-training rewards.