Harness choices alter agent scores, tool habits, and security outcomes

AgentCompass separates benchmarks, harnesses, and execution environments, then shows that the same model’s result can vary with the harness. On SWE-bench-Pro, Claude-Opus-4.8 scored 66.21 with Mini-SWE-agent and 73.87 with OpenHands. A separate set-shifting benchmark finds that agents quickly settle into recurring tool routines and may keep using an unreliable group after a silent backend change. Together, the studies make interaction history and harness configuration part of the measured capability, rather than incidental test plumbing.

Security agents benefit when exploration is driven by a concrete hypothesis. DREA’s planner requests repository evidence from a lightweight explorer; pair-level correctness rose from 19% to 42% for DeepSeek-V3.2 while more than 93% of tokens were handled locally. ProfMalPlus similarly routes uncertain package judgments toward static, registry, or sandbox evidence, reporting 98.1% F1 and 597 newly identified malicious NPM packages. VisualRepair applies the same selective-attention principle to screenshots, resolving 196 SWE-bench Multimodal test issues—10 more than its best baseline. The shared result is conditional: narrower evidence gathering helps, but DREA still found flawed rationales behind 26–55% of true positives.

Real-world adoption still centers human oversight. Across 2,361 GitHub repositories, the median project produced only one or two agentic pull requests in three months; one-person workflows accounted for 88.7% of observed cases. Governance proposals are appearing alongside this limited deployment base. EBAE separates an agent’s proposal from protected execution through action-specific, epoch-bound authorization, while DNSid links agent identity to internet domains and cryptographic registration records. Both mechanisms remain lightly validated: EBAE provides no quantitative evaluation, and DNSid reports unnamed trials without results.