Coding Agent Control Points

Teams evaluating coding agents should score the harness as a product component, not as background plumbing. Claw-SWE-Bench shows why: with the same GLM 5.1 model, OpenClaw scored 19.1% Pass@1 with a minimal direct-diff adapter and 73.4% with the full adapter. Under fixed models, harness choice changed Pass@1 by as much as 27.4 percentage points.

A practical adoption step is to add a small fixed benchmark run to every harness change. The run should keep the task set, Docker workspace, prompt template, wall-clock budget, patch extraction, and prediction format stable, then report Pass@1 beside token cost and wall-clock time. Claw-SWE-Bench Lite is a useful pattern because its 80-instance subset tracked the 350-instance run closely while costing about 22.9% of the full run. This gives engineering teams a way to catch regressions in adapters, stopping rules, and patch extraction before attributing movement to the model.

Agent-authored pull requests need a checkpoint that catches repeated failed fixes and unsupported claims of completion. PROJECTMEM supplies the state layer: it records issues, attempts, fixes, decisions, and notes in an append-only plain-text log, then warns through precheck_file(path) before edits to files tied to failed attempts, open issues, or high churn. agent-gate supplies the completion layer: verify_gate(...) fails closed unless required evidence fields are explicitly true.

A buildable workflow is a pre-merge bot for AI-authored changes. Before the agent opens or updates a pull request, it checks touched files against the project memory log. Before the PR is marked ready, it requires deterministic checks, independent refute-first review, a secrets check, human approval for irreversible or outward actions, and a receipt in a hash-chained ledger. The first useful measurement is simple: count how often the bot blocks repeated edits to known-problem files or blocks a “done” claim with missing tests, review, secret scan, approval, or receipt.

Teams enabling grammar-constrained decoding for code should add a security regression suite for malicious code prompts. CodeSpear shows that a normal programming-language grammar can remove natural-language refusals from the valid output space and leave the model sampling only syntactically valid code. The reported attack reached 81.82% average success on local models such as Qwen2.5-Coder-7B and beat representative jailbreak baselines by more than 30 percentage points on average across tested models.

The concrete test is to run a malicious-code benchmark with and without the grammar constraint for each model and inference stack. The gate should fail when the constrained run produces executable harmful code at a higher rate than the unconstrained run. If grammar output is mandatory, CodeShield points to one mitigation pattern: train preference behavior for cases where only code is allowed, so the model emits harmless code when a refusal cannot be represented.