Code agents are being graded on complete, verifiable work

SWE-Cycle makes the gap visible between partial progress and full issue resolution. It asks agents to rebuild the environment, implement the fix, and write verification tests in a raw repository. The best FullCycle solve rate in the excerpt is 13.50%, even though isolated environment reconstruction reaches 78.12% and isolated test generation reaches 67.28%.

AI Harness Engineering gives a systems explanation for this pattern. It treats agent performance as the result of the model, runtime harness, and repository environment together. Its proposed trace package records actions, tool use, context, verification, failure attribution, intervention, entropy, and outcome, so a successful patch must come with evidence about how it was produced.

Phoenix-bench tests software-style agents on Verilog and SystemVerilog maintenance tasks under executable Electronic Design Automation (EDA) checks. Top commercial agents resolve only 32.7% to 38.6% of tasks. The same agents lose 37 to 58 percentage points compared with SWE-bench Verified.

The failure mode is concrete. Hardware bugs propagate through ports, clocks, resets, parameters, and signal flow across instantiated modules. A file-level oracle adds only 1.4 percentage points because agents still make wrong module edits. One round of testbench-log feedback raises resolved rates to about 42% to 45%, showing that executable failure evidence is more useful than file names alone.

PBT-Bench isolates property-based testing (PBT): the agent must infer a documented invariant and write Hypothesis input generators that reach semantic bugs. The benchmark covers 100 problems, 40 Python libraries, and 365 injected bugs. Under the PBT-guided prompt, recall spans 42.1% to 83.4% across models.

The prompt gains are uneven. Qwen 3.6 Plus gains 24.5 percentage points, Qwen 3.5-30B-A3B gains 22.9 points, and Step 3.5 Flash gains 20.3 points. DeepSeek V3.2 and Grok 4.1 Fast lose recall under the same scaffolding. This makes test-design competence a measurable agent skill, with model-specific prompt fit.

CRANE edits Qwen Instruct checkpoints with reasoning directions from Thinking checkpoints, then suppresses updates that would disturb tool delimiters, JSON schema tokens, and chat templates. On Roo-Eval, it reaches 66.2% pass@1 at 30B compared with 46.7% for the Instruct baseline, and 81.5% at 80B compared with 72.8%.

Other work tunes the surrounding machinery. CANTANTE assigns per-agent prompt credit from system-level scores and reports the best average rank across MBPP, GSM8K, and HotpotQA. SkillOps maintains reusable agent skills as typed contracts with validators and compatibility links, reaching 79.5% task success on ALFWorld with a 200-skill library. 1Password’s monolith refactor shows the production version of the same lesson: agents helped after engineers supplied deterministic analyzers, manifests, stop rules, and review points.