Coding Agent Acceptance Controls

Teams using coding agents can make repository guidance part of the test workflow. A practical version starts with a short AGENTS.md file, generates synthetic bug-fix probes for that repository, asks the agent to attempt fixes, and edits the guidance only where the failed runs show missing or misleading instructions. The resulting file should name subsystem entry points, test commands, known brittle paths, and quality gates in compact form.

Probe-and-Refine gives a concrete shape to this workflow. On 500 SWE-bench Verified instances, refined guidance reached a 33.0% mean resolve rate, compared with 28.3% for a static knowledge base and 25.5% with no context. The reported gain came mainly from more evaluable patches, not higher per-patch precision. That matters for maintainers because a bad instruction file often sends an agent to the wrong file or leaves it stuck before a meaningful patch exists.

A cheap adoption test is to run the same agent on a small set of recent fixed bugs with three guidance variants: no file, a hand-written file, and a probe-refined file. Track whether the agent finds the right files, runs the right tests, and produces a patch that reaches review. If the refined file only adds generic advice, it should be rejected like any other unhelpful repository artifact.

Agent-generated pull requests need a small action log beside the diff. The useful record is simple: files read, repository searches, edits, commands, tests, retries, and submission. A CI gate can then check rules such as “tests ran after the final edit,” “the agent inspected the touched subsystem before editing,” and “the task did not submit after repeated test failures.” These checks give reviewers evidence about the work path before they spend time reading the patch.

ProcGrep shows why this is practical. It turns coding-agent trajectories into action sequences such as read_file, search_repo, edit, run_test, and submit, then supports deterministic structural queries over those traces. On SWE-bench Verified traces from 10 agents, procedural fingerprints attributed unseen trajectories to the correct agent with 85.7% accuracy, and ProcGrep reached F1=1.000 on an episodic trace-search task at microsecond latency per decision.

GlueRun-go points to an implementation pattern for teams already running parallel agents. Each worker runs in a separate Git worktree, holds a JSON lease, writes a state packet with changed files, commands, tests, and evidence, and sends gate results into an audit and recovery path. The project evidence is engineering evidence, not a standard benchmark, but the shape is useful: isolate the work, record the task evidence, and make gate failure a first-class outcome.

Before giving an agent a new codebase, internal library, or private documentation set, teams can run a short open-book corpus exam. The agent receives the same corpus it will use on the real task, then answers hidden questions under several token budgets. Useful questions ask where behavior is implemented, which API is current, what configuration is unsafe, and which docs support the answer. The score should reward accurate answers at lower inference cost, because brute-force search is a poor operating mode for routine engineering work.

StudyBench makes this evaluation concrete. It pairs a corpus with a hidden exam across DSPy code, OpenClaw code, and recent machine-learning literature, then measures performance as inference tokens increase. The reported DSPy result for Qwen3.5-9B is a warning sign for agent deployment: forcing 20 search iterations raised the lenient score from 9.6 to 29.4, which means relevant evidence can be present and still go unused without better study behavior. On OpenClaw, two frontier models stayed barely above 10% even with more search.

A team can start with 20 to 40 private questions drawn from recent incidents, API migrations, and maintainer review comments. Passing the exam should require cited file paths or document references. Failed answers are useful even when the agent later writes code, because they identify the missing notes, stale docs, and retrieval paths that will cause bad patches.