Code agents are being judged by delivered systems and verified repair loops

SaaSBench makes enterprise software delivery the test. Its tasks include long product requirements, Docker runtimes, dependency-ordered validation nodes, multiple languages, databases, and frontend/backend stacks. The best reported Pass@1 is 20.68%, and over 95% of failures happen before deep business logic, often during setup, configuration, integration, premature stopping, or stalled debugging.

WebGameBench gives the same point a user-visible form. Agents build browser games, then a runtime evaluator controls Chrome through Playwright and checks behavior against the requirement. The best configuration reaches 76.9% usable rate, but only 20.2% excellent rate. A page can load and still miss rules, input handling, scoring, restart behavior, or win/loss conditions.

Two repair papers make vulnerability fixing a feedback problem. ContraFix compares crashing and safe executions, inserts probes near the fault, and writes a repair specification before patching. On SEC-Bench it resolves 84.0% of 200 C/C++ CVE instances, and its ablation credits contrastive runtime analysis with a 27-point gain.

MemRepair takes a memory-centered route. It stores past fixes, reusable security patterns, and failed-patch-to-success trajectories. A Locator, Patcher, and Verifier loop runs vulnerability and regression tests before accepting edits. On SEC-Bench, MemRepair with DeepSeek-v3.2 resolves 58.00% of tasks, ahead of the listed OpenHands, SWE-agent, and Aider baselines.

FireFly builds tool-use training data by executing real Model Context Protocol (MCP) APIs first and writing tasks after observed outputs exist. The pipeline filters Smithery servers to 240 servers and 993 tools, builds a directed tool graph, explores live APIs, and caches observed calls for offline reinforcement learning.

The result is a dataset of 5,144 verified tasks and 9,749 trajectories. Qwen3-4B improves from 28.1% to 41.5% pass@1 on the FireFly test after training, close to Claude Sonnet 4.6 at 42.2%. The paper also reports gains on Tau2-Bench, MCP-Atlas, and MCPMark.

The code-review paper treats agentic review as a coordinated pull request process with human decision points. Its five stages cover PR creation, PR augmentation, reviewer selection, AI-assisted review, and retrospective. The paper is a vision piece: it reports no new benchmark, prototype evaluation, user study, or controlled experiment.

AgentModernize applies a more testable structure to legacy modernization. It extracts business rules into a Behavioral Specification Graph, generates Python/FastAPI code, and validates behavior through feedback. The results stay low: GPT-5.3-codex reaches 19.4% mean Behavioral Equivalence Rate, while single-prompt and chain-of-thought baselines score 0.0% across tested scenarios.