Inspectable Agent Operations

Benchmark maintainers can add a pre-release red-team pass that tries to earn task credit without doing the task, records the exploit path, and reruns after fixes. BenchJack gives a concrete pattern: map entry points, scoring code, task files, environments, and trust boundaries; scan against a flaw taxonomy; then generate a tested exploit such as a run.sh that maximizes score. In its reported audit, BenchJack produced working exploits for all 10 agent benchmarks it tested and found 219 distinct flaws. Its patching loop cut hackable-task ratios below 10% on four fixable benchmarks and fully patched WebArena and OSWorld within three iterations.

The release checklist should also include rollout-card exports for the same benchmark runs. Rollout Cards stores the task, environment state, observations, model outputs, tool calls, tool results, artifacts, timing, terminal status, failures, declared scoring view, reporting rule, and omitted fields. The paper’s audit of 50 repositories found that none reported failed, errored, or skipped rollouts beside headline scores, and re-grading fixed artifacts changed reported scores by up to 20.9 percentage points. A benchmark release can treat an unpublished exploit scan and missing rollout records as release blockers, because both affect whether later users can trust the reported score.

Enterprise AI owners can move MCP servers off employee laptops and place them behind a managed approval, identity, logging, and policy layer. Cloudflare’s internal deployment gives a concrete operating model: employees expose an internal resource through a template after approval, inherit default-deny write controls, audit logging, CI/CD, and secrets management, and use OAuth through Cloudflare Access with SSO, MFA, IP, location, and device checks. Its portal pattern gives each employee one endpoint that exposes only authorized MCP servers, while traffic scans look for shadow MCP use through /mcp, /mcp/sse, and JSON-RPC methods such as tools/call, tools/list, and resources/read.

The trace requirement should be explicit at the point of MCP approval. A pilot on agent decision reconstructability found strict governance completeness ranging from 42.9% to 85.7% across six SDK regimes, with reasoning evidence missing or unusable across most surveyed regimes. A useful first rollout is one high-risk internal tool with writes disabled by default, per-user authorization, DLP rules, and a stored decision record for each tool call: actor, policy, tool, arguments, result, authorization source, and available reasoning evidence. Security teams then get an incident record that can answer who authorized a tool action and which policy applied.

Teams translating legacy C or C++ into Rust can add a review gate that shows developers only the behavioral differences the tool can prove are reachable. cozy compiles the original and translated programs to binaries, runs both under symbolic execution with the same symbolic inputs, compares compatible terminal states, and uses Z3 to prove selected outputs or generate concrete inputs that expose a difference. The developer then classifies each flagged difference as intended or erroneous, while the unflagged paths are treated as equivalent under the checked bounds.

This is a practical fit for incremental memory-safety migrations where automated translators, manual ports, and bug fixes can all change behavior. The reported cozy experiments are small: insertion sort, a watch update function, and a box blur filter. Even so, the workflow is concrete enough for a pilot on a bounded utility function with clear inputs and outputs. APL-to-C# work in the pack points to the same adoption pattern for other legacy languages: generate typed targets, compile them, run input-output tests, and feed compiler or test failures into repair attempts. The main gap is scale, so the first adoption target should be a small module with existing tests and well-defined state.