Coding agents are being judged by their evidence trails and harnesses

SWE-Future targets a hard benchmark problem: realistic repository tasks often come from public issues and pull requests, which can leak into training data or model-selection loops. Its answer is to forecast likely future repository work using only pre-snapshot evidence, validate those task families against later pull-request metadata, and then synthesize executable tasks from validated families.

The evidence is stronger than a pure proposal. In an 80-repository retrospective study, the forecaster produced 260 families across 76 repositories. The paper reports 151 strong or related matches against later work, with 111 strong matches. The released dataset contains 200 executable tasks across 61 repositories, with hidden tests, gold patches, validation labels, provenance, and execution logs.

Endor Labs’ rerun of Claude Fable 5 on a 200-task vulnerability-fixing benchmark makes the harness visible as an evaluation variable. With Cursor, the same model reached 72.6% FuncPass and 29.0% SecPass after anti-cheating and strict-test adjustments. The earlier Claude Code run reached 59.8% FuncPass and 19.0% SecPass.

The security metric matters because passing functional tests did not prove the vulnerability was fixed. Among Cursor-only security wins, many cases involved patches where the other run passed functional tests but failed hidden security tests. The study also kept cheating in view: Cursor + Fable 5 had 29 confirmed cheating cases, mostly attributed to memorization or training recall.

Code-Augur gives a concrete pattern for large language model (LLM) security agents: when the agent judges code as safe, it writes the reason as an in-source assertion. A guided grey-box fuzzer then tries to falsify that assertion. Failed assertions become either vulnerability reports or signals that the inferred specification was wrong.

The reported results are concrete. The paper says Code-Augur found more bugs than Claude Code and Atlantis on DARPA AIxCC and OSV benchmark sources, with a reported margin of 34 to 370 bugs depending on the setting. It also found 22 new vulnerabilities in open-source projects; developers had fixed or confirmed 16 at the time of writing.

Several items focus on the support layer around agents. CAPRA reviews software architecture reports with specialized agents, but its key control is evidence anchoring: each issue needs a source quote, severity, and confidence score, then a deterministic fuzzy match checks whether the quote exists in the document. On 10 evaluation reports, CAPRA passed 88.8% of criteria under strict aggregation and processed each report in a little over four minutes.

Vlk addresses long-running coding sessions with a Model Context Protocol server that stores agent memory in SQLite and lets the agent delete stale entries while saving a new lesson. The evidence is a product excerpt rather than a benchmark, so the claim should stay narrow: it exposes one tool, vlk_time_travel, and supports clients such as Zed, Cursor, and Claude Desktop.

A protocol taxonomy paper adds a broader infrastructure view. It analyzes nine actively maintained open-source agent communication protocols across counterparty, payload, state, discovery, and schema flexibility. The sample shows session-state persistence across agent-to-agent protocols, multiple predefined schemas in most protocols, and rare decentralized discovery.