Executable gates expose failures that static agent scores miss

Reliable automation increasingly depends on deterministic controls outside the model. A setup-security study found that agents usually missed malicious package sources across ecosystems; prompts only helped for the attack dimension they named, while a pre-install check covering names, sources, and versions closed most of the observed gap. Proof-or-Stop applies the same principle to lifecycle decisions: claims such as “tested” or “done” advance only with fresh evidence bound to the tracked source state. Its gated loop recorded zero false-done outcomes in 10 scenarios and rejected all 18 tested receipt-tampering classes, though evaluation was limited to one model family and a self-hosted corpus.

Benchmarks are treating the full executable workflow—not plausible code or a final answer—as the unit of success. Alipay-PIBench tests payment integration with end-to-end behavior, signature and notification handling, refund safeguards, and business-state consistency; supplying an official domain skill raised mean rubric pass rate by 10.31 percentage points. StructureClaw requires a linked chain of models, validation records, solver outputs, and reports, raising average success from 56.8% to 88.6% with its governed workflow. Kaleidoscope extends the pattern to deployed applications by calibrating application-specific judges against human labels, but its evidence remains an uncontrolled four-use-case pilot.

Controlled evaluations show that capability measured in a fixed, text-dominant environment does not transfer cleanly. On evolved Model Context Protocol (MCP) servers, GPT-5.4 and Claude-Sonnet-4-6 lost 13.7% and 14.4% task performance. In repository localization with screenshots and other visual evidence, the strongest agent reached only 38.96 file Acc@5 and 22.45 function Acc@10. Maintenance evidence points to a related scope problem: 28 of 64 sampled AI/ML issues required changes beyond production code, including prompts, datasets, dependencies, and runtime configuration. Evaluation therefore needs to preserve environmental change and heterogeneous artifacts rather than reducing work to a static code snapshot.