Coding Agent Verification Trails

Teams that use coding agents to write tests can add a small mutation gate before those tests affect patch approval, benchmark scores, or repair-agent training. A generated test suite should fail on the original bug, pass on the fix, and catch a few semantically plausible faulty variants derived from the fix.

SWE-Mutation shows why this gate is useful. DeepSeek-V3.1 with Mini-Swe-Agent reached 88.20% Pass@1 on Python test generation, yet only 10.20% verification and 36.15% mutant detection. The test files ran, but many did not prove the reported issue or catch nearby wrong repairs. A cheap pilot is to take recent fixed issues, generate 3 to 5 mutants per fix, and track VRR and RDR beside Pass@1 in CI or evaluation runs.

Repositories accepting agent-authored pull requests should record review evidence in the same place they record merge status. A GitHub bot can attach fields for CI result, reviewer comments, reviewer-applied commits, closure reason, duplicate or superseded work, and explicit human feedback. For Java patches, the same check can run RefactoringMiner and flag tangled refactorings that were not requested by the issue.

The pull-request study found that only 35.7% of sampled rejected agentic PRs were clear agent failures, while 31.2% came from workflow constraints and 33.1% had no visible rationale. Merged PRs also need context: 15.4% involved explicit reviewer feedback or reviewer-applied commits. Refactoring Runaway adds a patch-level reason to keep this record: tangled refactorings appeared in 21.43% of valid Java agent patches and were associated with lower compilability. RefUntangle raised average compilation success from 19.34% to 38.33%, which points to a practical pre-review repair step for risky structural edits.

Teams running deployed agents can turn repeated user-facing failures into a guarded update workflow. The system logs weak or missing dialogue chunks, seals a small batch, asks a coding agent to locate and edit the source-level fault, and replays the batch against a candidate image in an isolated worker before any production change. Promotion requires a human apply step, health probes, and rollback.

MOSS gives a concrete pattern for this workflow. It targets failures in routing, hook ordering, session state, dispatch, and concurrency, which prompt or skill edits may not fix. On OpenClaw, one evolution cycle raised the four-task mean grader score from 0.25 to 0.61 without human code edits. The safety detail matters for adoption: MOSS tests candidate images in ephemeral trial-worker containers and requires moss evo apply, health checks, and rollback before promotion.