Provenance-Aware Coding Agent Evaluation

Security teams evaluating coding agents should score the model and the agent harness as a pair. Endor Labs reran Claude Fable 5 on the same 200 vulnerability-fixing tasks through Cursor and compared it with an earlier Claude Code run. Cursor + Fable 5 reached 72.6% FuncPass and 29.0% SecPass after anti-cheating and strict-test adjustments, while Claude Code + Fable 5 reached 59.8% FuncPass and 19.0% SecPass.

The workflow change is concrete: keep a fixed set of real vulnerability-fix tasks, run the same model through each candidate IDE or CLI harness, and report FuncPass, SecPass, timeouts, empty patches, and confirmed cheating separately. The SecPass split matters because a patch can pass visible functional tests and still leave the vulnerability open. In the Endor Labs comparison, 13 of 25 Cursor-only security wins were cases where the Claude Code run passed functional tests but failed hidden security tests.

A cheap adoption check is a 20-task slice from the organization’s own past vulnerability fixes. If the same model produces different SecPass results across Cursor, Claude Code, and any internal harness, procurement and AppSec teams get a more useful buying signal than a model-only leaderboard score.

LLM security audit agents should write their safety assumptions as executable assertions at the code site they are judging. Code-Augur follows this pattern: the agent builds a threat model, records local invariants as in-source assertions when it thinks code is safe, and sends the instrumented program to a guided grey-box fuzzer. A failed assertion becomes either a vulnerability report or a bad specification that the agent must refine.

This gives security engineers a direct artifact to inspect after an agent says a file is safe. The surviving assertions can also be kept with the audit record, so later reviews can retest the same assumptions when nearby code changes. The paper reports that Code-Augur found 22 new vulnerabilities in open-source projects, with 16 fixed or confirmed by developers at the time of writing.

A first implementation can target parsers, image codecs, protocol handlers, and other input-heavy code. Ask the agent to emit assertions only for attacker-controlled input boundaries and state conversions, run existing fuzz targets or a short fuzzing budget, and triage assertion failures before filing bugs.

Benchmark maintainers can reduce direct replay of public pull requests by generating tasks from repository signals available before a cutoff date. SWE-Future builds evidence bundles from pre-snapshot issues, pull requests, labels, titles, and short text, forecasts task families, freezes those forecasts, and then validates them against later pull-request metadata. The later patches are used for validation, not as task prompts or gold solutions.

The reported dataset shows enough detail for a small internal trial. In an 80-repository retrospective study, SWE-Future produced 260 task families across 76 repositories. The final release contains 200 executable tasks across 61 repositories, with hidden tests, gold patches, validation labels, provenance, and execution logs. Bugfix forecasts had the clearest signal, with 89 of 139 bugfix families matching later work as strong or related.

An evaluation owner could pilot this on 10 active repositories: freeze a snapshot, generate bugfix and enhancement families from earlier issue and PR text, wait for a validation window, then synthesize executable tasks only from families that match later project work. The resulting benchmark carries a provenance trail and a time boundary that are easier to audit than copied public issue-and-PR pairs.