Agent Execution Integrity

Security teams should add a workflow check that traces whether attacker-controlled fields, such as GitHub issue comments, can reach an agent prompt that also has secrets, shell access, API tokens, or database tools. The useful output is a small path report: trigger event, transformed prompt fields, agent call, available tools, and the action that could leak data or execute commands.

JAW gives a concrete recipe for this check. It combines workflow path analysis, runtime prompt tracing with canary-marked inputs, capability checks, and payload evolution. In real GitHub Actions and n8n templates, it found 4,174 hijackable GitHub workflows and eight hijackable n8n templates, with reported impacts including credential leakage and arbitrary command execution. A first internal version can run on pull requests that change workflow YAML, reusable actions, n8n templates, or agent permissions, then block only the cases where untrusted prompt content and privileged tools meet in the same execution path.

Coding-agent runners should record model calls, tool calls, file writes, and environment actions as a typed execution trace that a reviewer or supervising agent can inspect and fork. The practical feature is a “resume from here” control on failed or risky runs: keep the exact prior state, branch before the bad command, try a different continuation, and preserve the trace for review.

Shepherd shows that this can be fast enough for live use. On Terminal-Bench 2.0 images up to 5.8 GB, it reports 134–143 ms forks, while full filesystem copy reaches 53,462 ms on the largest image. Replay also reaches about a 95% prompt-cache hit rate on Claude Haiku 4.5 across eight tasks. The same paper reports a live supervisor raising CooperBench pair-coding pass rate from 28.8% to 54.7%. A small adoption test is to wrap one internal coding-agent runner, record every filesystem and tool effect, and measure how often reviewers can repair a failed run by branching before the first wrong state change.

Teams exposing many tools through MCP should test agents against seeded stateful sandboxes and score the final environment state, not only whether the agent produced a plausible answer. The regression suite should include changing permissions, carts, chat histories, accounts, API failures, and nested state. Each task should report completion, misbehavior, skipped checks, and recovery after tool errors.

ComplexMCP is a useful template because it tests more than 300 tools across seven stateful sandboxes and uses rule-based comparison between the agent’s final nested environment state and the ground truth. The best reported model, Gemini-3-Flash, reaches 55.31% success, while humans using the same interface reach 93.61%. The failure modes are directly actionable for product tests: tool retrieval saturation, skipped environment checks, and failure rationalization after errors. A cheap check is to convert ten high-value customer workflows into seeded MCP tasks and fail releases when a model changes the wrong state or stops after a recoverable tool error.