Coding Agent Release Controls

Repository teams using coding agents should add a small tuning job for their AGENTS.md or equivalent repository guidance. The job can generate synthetic bug-fix tasks, run the agent against them, record where the instructions sent it to the wrong subsystem or missed the right test command, then edit the guidance file under a size cap.

Probe-and-Refine gives a concrete pattern: 10 synthetic bug-fix probes per iteration, 3 to 5 iterations per repository, and a final guidance file reused by the coding agent. On 500 SWE-bench Verified instances, the refined guidance reached a 33.0% mean resolve rate, compared with 28.3% for static guidance and 25.5% with no context. The gain came mainly from producing evaluable patches more often, which fits the common maintainer pain: the agent wastes its budget in the wrong files or follows stale local habits.

A cheap first version is a nightly job on the top repositories where agents already open drafts. Track whether the agent reaches an evaluable patch, which commands it ran, and which guidance lines were used or contradicted. Keep the output short enough for every run to read, and require a human review for guidance edits that change release, security, or data-migration steps.

Teams letting agents work GitHub issues should put the agent behind a pull request lane that compares a clean baseline test run with the patched run. The gate should allow drafts only when the patch adds no new failing tests, blocks sensitive file paths or workflow edits, and records the agent’s tool calls, approvals, and owner for review.

Phoenix is a useful reference design because it handles broken or flaky repositories by separating pre-existing failures from failures introduced by the patch. Its Tester runs the unmodified branch first, runs the patched branch next, and accepts a change only when no new failures appear. On a curated 24-task SWE-bench Lite slice, Phoenix reports 18 oracle-resolved tasks and no PASS_TO_PASS regressions on successful runs.

Permission choice needs its own check in the same lane. ToolPrivBench found that six of 11 evaluated models exceeded 30% over-privileged tool use when lower-privilege tools were sufficient. Eve shows the production plumbing that makes these gates easier to ship: durable sessions, sandboxes, human approvals, traces, and file-based evals. A production repository can start with three required records on every agent PR: baseline test hash, patched test hash, and any high-privilege tool or approval event.

Engineering groups should stop accepting a single Python coding score as evidence for repository-wide agent use. A rollout eval should include the project’s main languages, compile or runtime checks in sandboxed containers, and at least one multi-file task with behavior checks for repositories where unit tests do not capture the main failure.

Multi-LCB gives the cross-language template. It extends LiveCodeBench to 12 languages while keeping release-date filtering and hidden tests. The reported scores show why this matters: Qwen3-235B-A22B-Thinking-2507 scores higher than GPT-OSS-120B Medium in Python and C++, then drops in Rust, Ruby, and Go, lowering its 12-language average.

JAMER adds the project-scale test. Its Godot benchmark filters more than 240,000 candidate repositories down to 8,133 behavior-valid projects and checks compilation, startup stability, structural completeness, and runtime behavior. Runtime pass rates on code completion fall from 80.4% on small projects to 5.7% on large projects. A practical adoption test can be small: pick 20 tasks from the team’s own languages and 5 large multi-file tasks, then block general rollout until the chosen agent passes the same CI and runtime checks a developer would face.