Agent verification records

Software teams adopting code agents should require each agent PR to include a compact trace package: environment setup steps, files inspected, commands run, generated or changed tests, verification results, failure diagnosis, and any human intervention. This gives reviewers a stable artifact to inspect before they spend time reading a large patch.

SWE-Cycle shows why this matters. Agents can do much better on isolated setup and test-generation tasks than on full issue resolution from a raw repository; the best FullCycle solve rate in the excerpt is 13.50%, even though isolated environment reconstruction reaches 78.12% and isolated verification test generation reaches 67.28%. AI Harness Engineering gives the operational checklist for the missing evidence, including action, tool, context, verification, failure attribution, intervention, entropy, and outcome traces. 1Password’s monolith refactor shows the production version: agents worked better when engineers built deterministic analyzers, manifests, stop rules, and review points, while sequencing mistakes still required human control.

Hardware teams testing software-style coding agents on Verilog or SystemVerilog should add an EDA feedback loop before broad adoption. A practical version runs the agent’s patch in a Docker-pinned Verilator or synthesis environment, returns the failing testbench log with module hierarchy context, and asks for one repair attempt in an isolated worktree.

Phoenix-bench reports that top commercial agents resolve only 32.7% to 38.6% of repository-level HDL issues under executable EDA checks, with drops of 37 to 58 percentage points compared with SWE-bench Verified. The failure mode is specific: agents stop at symptom files and miss signal-flow dependencies through ports, clocks, resets, parameters, and instantiated modules. File-level oracle localization adds only 1.4 percentage points, while one round of testbench-log feedback lifts resolved rates to about 42% to 45%. The cheap adoption test is a small internal benchmark of recent HDL bugs with fail-to-pass and pass-to-pass checks, scored only when the target failure passes and prior passing tests still pass.

Teams asking agents to write tests for Python libraries should trial property-based testing prompts per model before adding the workflow to CI. The concrete check is small: give the agent API documentation, existing tests, and a hidden buggy version, require a pbt_test.py file using Hypothesis, and score tests by fail-on-buggy and pass-on-fixed behavior.

PBT-Bench isolates this skill across 100 problems, 40 Python libraries, and 365 injected semantic bugs. Under a PBT-guided prompt, recall ranges from 42.1% to 83.4% across models. The same prompt helps some models by more than 20 percentage points and hurts others, including reported drops for DeepSeek V3.2 and Grok 4.1 Fast. A team can use this pattern to decide which model and prompt are allowed to generate property tests for documented invariants, especially for APIs where boundary cases and input distributions matter more than example-based tests.