--- kind: ideas granularity: week period_start: '2026-05-04T00:00:00' period_end: '2026-05-11T00:00:00' run_id: b344d1d3-282f-4658-b947-412e66d2fe3f status: succeeded topics: - coding agents - software engineering benchmarks - executable evaluation - formal verification - agent security - tool use tags: - recoleta/ideas - topic/coding-agents - topic/software-engineering-benchmarks - topic/executable-evaluation - topic/formal-verification - topic/agent-security - topic/tool-use language_code: en pass_output_id: 145 pass_kind: trend_ideas upstream_pass_output_id: 144 upstream_pass_kind: trend_synthesis --- # Coding Agent Trust Gates ## Summary Coding-agent adoption needs evidence checks inside normal engineering work: a pre-execution verifier for tool calls, repository acceptance rules for maintenance tasks, and staged-ticket security tests for backlog work. The common pressure is practical trust before live actions, merges, or security-sensitive deployments. ## Pre-execution contract checks for code-mode tool calls Teams giving coding agents access to APIs, deploy tools, ticket systems, or billing-sensitive services should add a verifier between generated code and the first live call. The verifier should read the task instruction and tool registry, check tool choice, output shape, call signatures, call ordering, and argument provenance, then return item-level repair feedback. RubricRefine reports 0.86 success on M3ToolEval across seven models, compared with 0.62 for single-pass CodeAct, with no environment execution during refinement. A practical internal test is 50 archived multi-tool tasks with known correct traces: measure wrong tool routing, malformed arguments, and provenance errors before and after the gate. The gate is most useful where a failed call changes external state, consumes quota, or creates an audit event. ### Sources - [RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement](../Inbox/2026-05-10--rubricrefine-improving-tool-use-agent-reliability-with-training-free-pre-execution-refinement.md): RubricRefine checks tool contracts before execution, covers tool choice, output shape, call signatures, ordering, and data provenance, and reports 0.86 average success on M3ToolEval versus 0.62 for CodeAct. - [RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement](../Inbox/2026-05-10--rubricrefine-improving-tool-use-agent-reliability-with-training-free-pre-execution-refinement.md): The paper’s abstract identifies inter-tool contract failures that can run to completion without runtime errors, which supports a pre-live-call verifier. ## Repository-agent acceptance checks for Q&A, tests, refactors, and no-change tickets Maintenance agents need PR acceptance rules that cover codebase understanding, test quality, refactor hygiene, and valid abstention. SWE Atlas evaluates 284 tasks across 18 active repositories and shows that regression tests can miss 15 to 40 points of refactoring quality; test-writing mutation checks miss 10 to 15 points of rubric quality. FixedBench adds the production case of stale reports: agents edited already-fixed code in 35% to 65% of cases under the main setting, and abstention prompts caused high wrong abstention on partially fixed tasks. A useful workflow is to require an evidence manifest with commands run, files inspected, tests added, behavior-preserving rationale for refactors, and a no-code-change option tied to reproduction and git-history evidence. Pilot it on recent duplicate tickets and partially fixed tickets before allowing autonomous merge rights. ### Sources - [SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution](../Inbox/2026-05-08--swe-atlas-benchmarking-coding-agents-beyond-issue-resolution.md): SWE Atlas covers codebase Q&A, test writing, and refactoring across 18 repositories, and reports gaps between functional checks and rubric checks for engineering quality. - [Coding Agents Don't Know When to Act](../Inbox/2026-05-08--coding-agents-don-t-know-when-to-act.md): FixedBench tests already-fixed issues and partial fixes, showing unwanted edits on fixed tasks and wrong abstention on partially fixed tasks. ## Staged-ticket security tests for coding-agent backlog work Security teams testing coding agents on direct harmful prompts are missing the Jira-style attack path shown by MOSAIC-Bench. It uses 199 three-ticket chains with deterministic Python proof-of-concept oracles on Docker deployments; nine production agents reached 53.3% to 85.9% end-to-end attack success with only two refusals. Add a regression suite that feeds an agent sequences of benign-looking tickets against a disposable app, composes diffs, and runs exploit oracles on the final deployment. Pair it with monitor tests using refined attack trajectories: MonitoringBench shows a full-trajectory Opus 4.5 monitor fell from 94.9% catch rate on elicited attacks to 60.3% on the best refined attacks. The first target is any team letting agents implement backlog items without a security engineer reviewing cumulative diffs. ### Sources - [MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents](../Inbox/2026-05-05--mosaic-bench-measuring-compositional-vulnerability-induction-in-coding-agents.md): MOSAIC-Bench evaluates staged three-ticket attack chains with deterministic proof-of-concept oracles and reports high end-to-end attack success across production coding agents. - [MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring](../Inbox/2026-05-10--monitoringbench-semi-automated-red-teaming-for-agent-monitoring.md): MonitoringBench shows stronger refined attack trajectories sharply reduce monitor catch rates, supporting refreshed monitor tests beyond direct elicitation.