Research idea

Verification changes for coding-agent planning, regeneration, and review

Day · 2026-07-14 · Software Intelligence

Coding-agent workflows should spend their verification budget where evidence is incomplete: expose behavior and state transitions to reviewers, test dependency replacements against counterexamples beyond the existing suite, and require distinct diagnostic evidence when several agents inspect the same repair.

3 ideas

Behavior-linked review packets for agent-harness changes

Maintainers reviewing changes to agent harnesses should receive a generated packet that names the affected runtime behavior, source locations, shared-state transitions, and checks that exercised them. Harness Handbook shows that behavior-centric navigation improves localization and scope control, especially for scattered and rarely executed paths; the large observational review study shows that faster agent participation can coincide with more review smells. The practical change is to route review by affected behavior rather than file ownership and let a fast review expand when a locator is stale, a state transition lacks a check, or verification fails. A pilot can compare these packets with ordinary agent summaries on historical harness pull requests, measuring missed implementation sites and reviewer corrections rather than approval time alone.

Counterexample contracts for dependency regeneration

Supply-chain engineers evaluating a locally regenerated dependency need tests for intended behavior that the repository’s current suite never exercises. Use-case-oriented regeneration preserved 99.8% of observed validation behavior, but its failures included edge cases, class identity, and deep framework integration; passing the existing checks therefore defines only an observed boundary. Before replacement, engineers could translate short behavioral assertions into several executable contracts, retain both likely-valid and likely-invalid interpretations, and ask for clarification using a distinguishing input when they disagree—the ambiguity-handling pattern evaluated by Monty. The cheapest check is to replay known dependency edge cases and mutation-generated counterexamples against the original package and replacement; disagreement identifies either an unsafe replacement or an underspecified intended use before the dependency is removed.

Evidence-partitioned agent review for automated repairs

Teams reviewing automatically generated bug fixes should assign agents different evidence rather than ask several agents to repeat the same diff review. CT-Repair’s static, dynamic, and hybrid diagnoses repaired 99 more Defects4J bugs in union than its strongest individual perspective, while the review study found that multi-agent participation was faster but generally carried higher quality risk than human-only review. A repair pipeline should therefore require each reviewer to submit a root-cause claim tied to distinct static, runtime, or combined evidence, merge duplicate hypotheses, and send the unresolved disagreements—not the full comment stream—to a human maintainer. Evaluate this against same-budget homogeneous reviewers on seeded defects, counting unique valid findings and escaped regressions as well as review duration.