Coding Agent Operational Controls

Coding-agent evaluations should include a replayed developer session before a model or agent policy ships. The test should start with an incomplete request, let the user simulator inspect the workspace, require the agent to revise its plan, and record final verifier status alongside user correction turns and forgotten requirements.

SWE-Together gives a usable measurement pattern: it scores final repository correctness and User Correction on 109 repository tasks rebuilt from real sessions. SWE-INTERACT shows why this belongs in release checks. On the same underlying tasks, Opus 4.8 fell from 50.7% single-turn resolve to 26.7% in the interactive setting, while GPT-5.5 fell from 48.0% to 24.7% and its cost rose from $2.78 to $9.84 per trial.

A small adoption test is practical: take 20 recent internal agent sessions with enough repository state to replay, hide the final requirement set inside a reviewer script or simulator, and compare candidate agents on pass rate, correction turns, user messages, cost, and failure labels. This exposes agents that can discover most goals but still drop requirements or ship implementation bugs.

Infrastructure teams running coding agents should track cost at the LLM step level, split into prefix tokens, append tokens, and output tokens. The useful dashboard is session-shaped: requests, LLM steps, tool calls, prefix-cache hits and misses, long tool calls, and per-request cost.

TraceLab makes the case concrete. Its trace covers 4,265 Claude Code and Codex sessions, 357,161 LLM steps, and 432,510 tool calls. The median step reads about 119K prefix tokens and writes 214 output tokens. Prefix tokens account for 52.56B of 54.90B input tokens and 59.5% of estimated API cost. The global prefix-cache hit rate is high at 95.7%, yet misses still cause 3.8x more prefilling than truly new input tokens.

A cheap first build is a log normalizer for local and hosted coding-agent sessions. It can drop raw user text and tool I/O, keep timestamps and token counts, and flag sessions where prefix-cache misses or one-minute-plus tool calls dominate cost. That gives product and infra teams a shared view of which agent loops are expensive before changing model choice or context policy.

Teams connecting agents to MCP tools should put enforcement outside the agent and check each call against an approved tool path and data-flow rule. The practical starting point is a gateway that blocks sessions with no installed commitment, records reason codes, and prevents sensitive reads from reaching external sinks through later tool calls.

trajeckt demonstrates the concrete mechanism for MCP-speaking agents: a sealed commitment graph declares allowed tools, order, data sinks, scopes, and budgets before execution. In its smoke test, read_database and summarize are allowed, then send_email_external is blocked with HTTP 403 when sensitive data would leave the boundary. The evaluation is still narrow, so teams should pilot it on a small set of high-risk workflows such as database lookup plus ticket update, email, or shell access.

Tool exposure also needs an operational limit. The MCP pattern study reports that tool-selection accuracy drops below 90% between 10 and 15 visible tools for Claude Haiku 4.5 and between 20 and 30 for Claude Sonnet 4. A gateway can combine both controls: expose only the tools relevant to the current session scope and enforce the approved call path after selection.