Coding Agent Audit Trails

Coding-agent teams should record the file paths and line ranges an agent inspected before it writes a patch, then score those regions against known fixes on a held-out issue set. SWE-Explore shows why this is a practical evaluation target: it asks explorers to return ranked code regions under a fixed line budget, with ground truth derived from successful repair trajectories. Its exploration metrics correlate with downstream repair rates, including Context Efficiency at Pearson r=0.950 and first useful hit at r=0.928.

A cheap internal test is to replay 50 to 100 closed issues, hide the final patch, and ask each agent to produce five ranked regions before editing. Review can then separate missed-code failures from patch-generation failures. This gives engineering managers and tool builders a clearer reason for bad runs than a single pass or fail result.

Benchmark maintainers and internal eval teams can add randomized accepted outputs to a subset of coding tasks, then flag scores that exceed the expected cap. CapCode defines the cap as B = 1/M when a task has M equally valid outputs and the evaluator samples one. A non-cheating agent cannot know the sampled value, so above-cap results become a statistical warning sign for leaked tests, hardcoded outputs, or grader-specific behavior.

This can start as a canary set inside an existing eval suite. The team keeps normal tests for capability measurement, adds capped variants for leakage detection, and runs a one-sided binomial test on repeated submissions. The useful output is not only a lower score; it is a warning that an agent’s eval gain may come from exploiting accessible feedback.

Organizations that allow Claude Code, OpenCode, Cursor, Gemini CLI, or similar tools to load third-party skills need a pre-install check that executes each skill in a sandbox and observes file, process, network, and tool behavior. MalSkillBench shows that skills combine markdown instructions, executable scripts, and agent tool use, so static code scanning alone can miss attacks that depend on the agent following instructions at runtime.

A practical gate is a small quarantine service for new SKILL.md packages. It runs the skill in Docker with a test agent, records system calls and file changes, and blocks skills that match known malicious behaviors or request unsafe tool permissions. The benchmark’s false-positive finding matters for adoption: high-recall transfer tools produced up to 3,979 false positives on 4,000 benign skills, so teams should measure both blocked attacks and developer disruption before enforcing a detector broadly.