Execution evidence tied to changing dependencies, tools, and artifacts

Agent-platform release engineers should make a successful workflow receipt identify not only the source commit, but also every package source and version and the exact MCP server schema used during execution. Package-install experiments found that agents commonly accepted untrusted registries, while MCPEvol-Bench measured 13.7% and 14.4% task-performance declines for two frontier models after tool evolution. Proof-or-Stop shows how lifecycle evidence can be rejected when it is stale or detached from tracked state, but its current evaluation does not establish coverage of changing external tools.

A release gate could replay a small set of high-value workflows against both pinned and candidate dependency/tool versions, run deterministic package provenance checks before installation, and issue a receipt over the code commit, registry identity, dependency lock, MCP schema, and test outputs. Any change to those inputs would invalidate the prior receipt. The cheapest useful check is to mutate one registry URL and one MCP parameter in an existing release suite and verify that the gate blocks reuse of the old evidence and localizes the failed step.

Maintainers of AI/ML repositories should prevent an issue from being marked resolved until the workflow records which prompts, datasets, model settings, dependencies, runtime configuration, and source files were inspected or changed, then attaches verification appropriate to each affected artifact. In a study of four AI/ML projects, 28 of 64 AI/ML issues required changes outside production code. StructureClaw separately found that requiring a linked, executable chain of models, validation records, solver outputs, checks, and reports raised mean workflow success from 56.8% to 88.6% in its structural-engineering benchmark.

The practical change is an issue template and closure gate that builds an artifact-impact manifest during diagnosis, carries artifact hashes and execution conditions into repeated or statistical tests, and refuses closure when an affected artifact has no current verification record. This extends executable evidence chains to maintenance work where a passing code test can coexist with an unchanged prompt, dataset, or runtime configuration. A low-cost pilot can apply the manifest retrospectively to recently closed model-behavior issues and measure how often the recorded fix omits an artifact later identified in the pull request or reproduction steps.

Payment SDK teams supplying coding-agent skills should release each skill with executable regression cases for signature verification, asynchronous notifications, idempotency, refunds, and business-state consistency, then rerun those cases across old and candidate API or MCP tool versions. Alipay-PIBench found that an official integration skill improved mean rubric pass rate by 10.31 percentage points, but MCPEvol-Bench shows that tool additions and modifications can disrupt previously successful agent plans. A skill can therefore improve initial implementation while silently becoming unsafe as the connected interface evolves.

The build change is to version the skill, tool schema, and domain rubric together. Deterministic end-to-end checks should control release decisions; any LLM-judged properties, such as product fit or semantic state consistency, should be calibrated against fresh human labels for the changed integration rather than inherited from the prior version. Kaleidoscope’s pilot is too small to validate this design broadly, but it provides a concrete calibration method and warns that combining several rubric dimensions in one judge prompt degrades performance.