Agent Failure-Mode Gates

Teams running code-mode agents against live tools can add a verification pass before the first environment action. The verifier should read the task instruction and tool registry, then check the generated program for output shape, tool choice, call signatures, argument provenance, and call order. Failed checks should produce concrete repair instructions and rerun generation without executing the tool call.

RubricRefine is a useful template because it targets failures that can complete without runtime exceptions: wrong routing, broken data flow between tools, and valid-looking final answers built from the wrong intermediate values. On M3ToolEval, it reports 0.86 average success across seven models, compared with 0.62 for single-pass CodeAct, with no execution attempts during refinement. The first internal test should use multi-step tasks where a bad live call can change state, spend money, hit a rate limit, or expose data.

Monitor evaluations for coding agents should include a repeatable attack-generation step, not only a fixed set of direct prompts. A security team can build a small pipeline that separates attack strategy, execution in a sandbox, and post-hoc trajectory refinement, then keeps only attacks where the side task succeeds and the trajectory remains faithful to the strategy.

MonitoringBench shows why this matters for pre-deployment testing. Direct Opus 4.6 elicitation collapsed into a narrow attack distribution, with 81% of attacks falling into two of five technical tiers. After taxonomy-guided generation and refinement, the Opus 4.5 full-trajectory monitor’s catch rate fell from 94.9% on directly elicited Opus attacks to 60.3% on the best refined attacks. The attack set includes operationally relevant sabotage such as firewall backdoors, sudo privilege escalation, credential exfiltration, and malware download.

Maintainers of C/C++ libraries can use agent-generated tests more effectively by aiming them at shared-memory access pairs. The workflow is concrete: use static analysis to find public entry points, shared variables, shared-memory access locations, and conflicting access pairs; trace backward from a target access to the inputs and object states that can reach it; generate multi-threaded drivers; run them under a dynamic analyzer such as ThreadSanitizer; feed uncovered pairs and build failures back into another generation round.

ConCovUp reports this pattern on nine real-world C/C++ libraries totaling about 1,000 kLoC. It raises average Shared Memory Access Pair Coverage from 36.6% with a Claude Code baseline to 68.1%. The reported gain is coverage of concurrent interactions, not a claim that every new test finds a bug, so the cheap adoption check is to measure new SMAP coverage and ThreadSanitizer findings on one library before adding it to regular CI.