Repository Agent Safety Gates

Teams adopting coding agents for repository work need a regression harness built from their own failure history, not only benchmark prompts. ABTest shows a practical recipe: take confirmed user-reported failures, abstract them into reusable workflow patterns and action types, then replay them as executable tests inside real repositories with expected file states and trace checks. The numbers justify treating this as a product requirement. From 400 confirmed failures, the paper generated 647 repository-grounded cases and surfaced 642 new true anomalies across Claude Code, Codex CLI, and Gemini CLI.

A concrete build is an internal replay suite that starts with issue tracker tickets, Slack incident reports, and support escalations for one coding agent deployment. The first version does not need model-based fuzzing. It can cover a narrow set of recurring workflow breaks such as editing the wrong file, leaving partial changes after rollback, claiming success with stale workspace state, or running the wrong command sequence. Run that suite on every agent upgrade, every tool-permission change, and every scaffold change. For teams already exposing agents to production repositories, this is a cheaper gate than waiting for user reports to pile up in live use.

Evaluation stacks for coding agents now need multi-PR sequences with persistent state and repository health checks. SWE-STEPS gives a clear reason. When evaluation moves from isolated pull requests to dependent sequences, reported success drops sharply: Claude Sonnet 4.5 falls from 66.25% to 43.75% on one split, and Gemini 3 Flash falls from 56.52% to 36.59%. The paper also reports worse repository health under agent-written code, including higher cognitive complexity and technical debt than human baselines.

A concrete workflow change is to add one stateful track beside existing single-task benchmarks. Start with a small chain of historical pull requests from one active repository, preserve workspace state across steps, and score both functional correctness and post-change maintainability signals such as static analysis warnings or complexity growth. This kind of test is useful for platform teams deciding whether an agent can handle backlog work over several days, and for buyers comparing agents whose leaderboard scores come from clean resets that remove the consequences of earlier mistakes.

Organizations loading third-party skills into coding agents need a skill-ingestion gate that inspects documentation examples and configuration templates before installation. The new supply-chain paper shows why this belongs in the setup path. Its attack hides malicious logic inside ordinary-looking skill documentation so the agent copies and executes it during normal setup or coding work. Across 1,070 adversarial skills, bypass rates reach 11.6% to 33.5% across four frameworks and five models. Static analysis blocks most attacks, but 2.5% still evade both static checks and model alignment, and the authors report four confirmed vulnerabilities.

A concrete build is a quarantine step for skills and MCP-style extensions: parse SKILL.md and related docs, extract executable snippets and config fragments, flag network destinations and shell commands, and require a clean isolated run before the skill can touch a developer workstation or CI secret scope. Teams can start with marketplace imports and high-permission internal skills. The immediate value is reducing a class of attacks that enters through examples and templates developers often treat as harmless reference text.