--- kind: ideas granularity: day period_start: '2026-06-29T00:00:00' period_end: '2026-06-30T00:00:00' run_id: ea97389e-c672-45bc-9fb1-d4cdf9480e80 status: succeeded topics: - coding agents - interactive benchmarks - long-horizon coding - LLM serving - agent security - MCP - software engineering evaluation tags: - recoleta/ideas - topic/coding-agents - topic/interactive-benchmarks - topic/long-horizon-coding - topic/llm-serving - topic/agent-security - topic/mcp - topic/software-engineering-evaluation language_code: en pass_output_id: 293 pass_kind: trend_ideas upstream_pass_output_id: 292 upstream_pass_kind: trend_synthesis --- # Coding Agent Operational Controls ## Summary Coding-agent teams can now add session-level checks to release and operations work: multi-turn tests that count user corrections, serving dashboards that show repeated prefix reads, and MCP gateways that block unsafe tool sequences before execution. ## Multi-turn release tests for coding-agent user burden Coding-agent evaluations should include a replayed developer session before a model or agent policy ships. The test should start with an incomplete request, let the user simulator inspect the workspace, require the agent to revise its plan, and record final verifier status alongside user correction turns and forgotten requirements. SWE-Together gives a usable measurement pattern: it scores final repository correctness and User Correction on 109 repository tasks rebuilt from real sessions. SWE-INTERACT shows why this belongs in release checks. On the same underlying tasks, Opus 4.8 fell from 50.7% single-turn resolve to 26.7% in the interactive setting, while GPT-5.5 fell from 48.0% to 24.7% and its cost rose from $2.78 to $9.84 per trial. A small adoption test is practical: take 20 recent internal agent sessions with enough repository state to replay, hide the final requirement set inside a reviewer script or simulator, and compare candidate agents on pass rate, correction turns, user messages, cost, and failure labels. This exposes agents that can discover most goals but still drop requirements or ship implementation bugs. ### Sources - [SWE-Together: Evaluating Coding Agents in Interactive User Sessions](../Inbox/2026-06-29--swe-together-evaluating-coding-agents-in-interactive-user-sessions.md): SWE-Together defines replayed multi-turn coding tasks, final correctness scoring, User Correction, and reported model results. - [SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions](../Inbox/2026-06-29--swe-interact-reimagining-swe-benchmarks-as-user-driven-long-horizon-coding-sessions.md): SWE-INTERACT reports large single-turn to interactive resolve drops, higher per-trial cost, and failure labels such as forgotten requirements and implementation bugs. ## Prefix-token cost dashboard for coding-agent serving Infrastructure teams running coding agents should track cost at the LLM step level, split into prefix tokens, append tokens, and output tokens. The useful dashboard is session-shaped: requests, LLM steps, tool calls, prefix-cache hits and misses, long tool calls, and per-request cost. TraceLab makes the case concrete. Its trace covers 4,265 Claude Code and Codex sessions, 357,161 LLM steps, and 432,510 tool calls. The median step reads about 119K prefix tokens and writes 214 output tokens. Prefix tokens account for 52.56B of 54.90B input tokens and 59.5% of estimated API cost. The global prefix-cache hit rate is high at 95.7%, yet misses still cause 3.8x more prefilling than truly new input tokens. A cheap first build is a log normalizer for local and hosted coding-agent sessions. It can drop raw user text and tool I/O, keep timestamps and token counts, and flag sessions where prefix-cache misses or one-minute-plus tool calls dominate cost. That gives product and infra teams a shared view of which agent loops are expensive before changing model choice or context policy. ### Sources - [TraceLab: Characterizing Coding Agent Workloads for LLM Serving](../Inbox/2026-06-29--tracelab-characterizing-coding-agent-workloads-for-llm-serving.md): TraceLab provides the step-level schema, token split, session scale, cache-hit result, and cost share for prefix tokens. ## MCP tool-flow enforcement with scoped tool exposure Teams connecting agents to MCP tools should put enforcement outside the agent and check each call against an approved tool path and data-flow rule. The practical starting point is a gateway that blocks sessions with no installed commitment, records reason codes, and prevents sensitive reads from reaching external sinks through later tool calls. trajeckt demonstrates the concrete mechanism for MCP-speaking agents: a sealed commitment graph declares allowed tools, order, data sinks, scopes, and budgets before execution. In its smoke test, `read_database` and `summarize` are allowed, then `send_email_external` is blocked with HTTP 403 when sensitive data would leave the boundary. The evaluation is still narrow, so teams should pilot it on a small set of high-risk workflows such as database lookup plus ticket update, email, or shell access. Tool exposure also needs an operational limit. The MCP pattern study reports that tool-selection accuracy drops below 90% between 10 and 15 visible tools for Claude Haiku 4.5 and between 20 and 30 for Claude Sonnet 4. A gateway can combine both controls: expose only the tools relevant to the current session scope and enforce the approved call path after selection. ### Sources - [Show HN: A Firewall for AI agents with auditing](../Inbox/2026-06-29--show-hn-a-firewall-for-ai-agents-with-auditing.md): trajeckt describes sealed pre-session commitments, fail-closed checks, taint tracking, audit reason codes, and the read/summarize/email block example. - [MCP Server Architecture Patterns for LLM-Integrated Applications](../Inbox/2026-06-29--mcp-server-architecture-patterns-for-llm-integrated-applications.md): The MCP server pattern study reports tool-selection degradation as visible tool count grows and recommends scoped aggregation or retrieval over tools for larger deployments.