Coding agents need traces, gates, and scoped memory to earn trust

Agent evaluation is getting more operational. ProcGrep turns coding-agent trajectories into action sequences such as file reads, repository searches, edits, tests, and submissions. On SWE-bench Verified traces, its procedural fingerprints attribute unseen trajectories to the correct agent with 85.7% accuracy, and its deterministic trace search beats LLM judges on the reported episodic-search task.

SWE-Future addresses another evaluation weakness: public GitHub issue and pull-request replay. It forecasts future repository task families from pre-snapshot evidence, validates them against later pull-request metadata, then synthesizes executable tasks. The daily trend evidence adds a practical warning: the same model can score differently under another agent harness, so benchmark claims need the harness in view.

Repair work is using runtime evidence to make agent edits easier to judge. PRACREPAIR adds execution traces and validation trace diffs to automated program repair. It reports 162 correct fixes on Defects4J V1.2 and 171 on V2.0 with GPT-4o, while keeping patch refinement tied to diagnostics rather than pass/fail feedback alone.

Repository guidance is also being tested against failures. Probe-and-Refine generates synthetic bug-fix probes, diagnoses where guidance failed, and edits compact repository instructions. On 500 SWE-bench Verified instances, it reports a 33.0% mean resolve rate versus 28.3% for a static knowledge base and 25.5% with no context. The gain comes mainly from agents reaching evaluable patches more often.

Tooling work treats agent action as something that needs custody. GlueRun-go runs parallel coding agents in separate Git worktrees, assigns JSON leases, records state packets, and feeds gate results into an audit and recovery path. Its public evidence is engineering-focused, with regression tests and claims about faster crash detection, not a standard benchmark result.

Memory systems are adding explicit sources and gap reporting. Vitrus stores company knowledge in Markdown, answers with citations, and reports stale, unsupported, contradicted, or missing information. Its API checks verify endpoint names, arguments, types, and permissions before calls run. The reported evaluations are mostly project gates and controlled tests, so the stronger claim is about design discipline, not broad generalization.

Machine Studying frames a hard evaluation question: can an agent read a new corpus before the exam and convert it into usable expertise? StudyBench tests this on DSPy code, OpenClaw code, and recent machine-learning literature. The metric rewards accuracy at lower inference-token budgets, so brute-force search is penalized.

The early results caution against treating search, long context, or simple fine-tuning as enough. On DSPy, forcing Qwen3.5-9B to use 20 search iterations raises the lenient score from 9.6 to 29.4, which shows that evidence can be present and still underused. On OpenClaw, two frontier models remain barely above 10% even with more search. The issue is becoming relevant for coding agents that must work inside unfamiliar codebases and private documentation.