Agent release checks tied to interaction history and executable evidence

Agent-platform teams operating redundant APIs should add silent backend shifts to release testing and run them across every supported harness. AgentCompass shows that changing the harness can move the same model’s SWE-bench-Pro score by several points and can expose different trajectory failures. The set-shifting benchmark shows a separate operational risk: after reliability changes during a persistent session, agents can remain locked into an obsolete tool routine even when an equivalent working tool is available. A release report should therefore include a model-by-harness matrix with post-shift tool shares, task completion, repeated calls, and recovery latency—not only aggregate task success. The cheapest check is to replay a small set of production-like sessions, silently fail the preferred backend halfway through, and compare recovery with a fresh-session control; this determines whether session history or the backend failure itself causes the loss.

Application-security teams should prevent an automated repository finding from blocking a merge until its claimed mechanism is supported by a code path, configuration, registry record, or sandbox trace. DREA improves vulnerability detection through hypothesis-directed repository exploration, yet 26–55% of true positives across evaluated systems had flawed rationales. ProfMalPlus supplies a practical escalation pattern: begin with static behavior graphs and source-preserving slices, then request registry or sandbox evidence only when the judgment remains unresolved. Applied to pull-request scanning, the detector would emit a provisional hypothesis, gather the cheapest evidence that could refute it, and attach the resulting slice or trace to the review. Test this first by relabeling a sample of current true-positive alerts for mechanism correctness and measuring how many merge-blocking findings would lose support after expert review.

Git hosting providers should test narrow, action-specific authorization inside the dominant one-person coding-agent workflow rather than begin with organization-wide autonomous permissions. In the observed GitHub sample, 88.7% of agentic pull-request workflows involved one human, and the median repository produced only one or two agentic pull requests in three months. EBAE provides a concrete execution rule for this setting: the agent may propose an action, but a protected executor releases it only when a one-use certificate, policy, freshness state, and exact intent agree within the same epoch. A GitHub App could turn a maintainer’s approval into a short-lived receipt limited to one repository, branch, commit range, and operation such as opening a pull request or rerunning CI. Because EBAE has no quantitative evaluation, the first useful test is operational: compare approval time, rejected stale actions, and maintainer overrides against the existing GitHub App permission flow on low-volume repositories.