Coding agents are being measured by state, recovery, and regression control

The strongest engineering result is i cat-agent. It splits GitHub issue resolution across an Explorer, Patch Editor, and Validator. The agents exchange structured events, and the Validator keeps tests and assertions hidden from the Patch Editor to reduce patch overfitting. On SWE-bench Pro, i cat-agent with GPT-5.4-xhigh solves 67.4% of tasks, 8.3 percentage points above mini-SWE-agent with the same backbone. The same paper reports lower average cost than Claude Code on SWE-bench Pro.

The broader design theme appears in Code as Agent Harness, a survey that treats code as the place where agents keep state, call tools, plan, and verify work. MCPlexer gives a concrete product version of that idea for the Model Context Protocol (MCP): a small shared tool surface with routing, approvals, audit logs, memory, browser control, and workspace policies. The MCPlexer item has no benchmark evidence, so its value is operational design, not measured performance.

ToolBench-X tests agents on 1,106 executable multi-step tasks with 4,956 Python tools and five recoverable hazard types. No evaluated model reaches 0.60 overall accuracy. The best reported score is 0.513 from Doubao-Seed-2.0-Lite. The paper’s diagnostic subset shows that targeted recovery hints regain 25.5 to 35.5 accuracy points, while extra interaction rounds help less. The weak point is hazard diagnosis and recovery choice.

CodeChat-Eval adds a separate pressure test for coding models: can they keep code correct through 10-turn refinement dialogues? Across 542 programming tasks, functional correctness drops for every evaluated model. GPT-5 Nano loses 19.2%, and Llama 3.1 8B loses 69.2% after multi-turn refinement. SWE-Pro makes the same accountability point for performance optimization. Expert patches produce large runtime and memory gains, while current large language models often apply patches but rarely deliver measured speed or memory improvements.

IntentTester shows a practical use of multi-agent coding beyond patch generation. It converts source tests into a language-neutral Test Description Language, maps those intents onto a target repository graph, then generates executable tests with verification feedback. This handles cases where API signatures and direct code patterns do not line up, including Java-Python migration.

The evaluation covers nine open-source projects across JSON, HTML, and Time libraries. From 2,058 source tests, the pipeline creates 5,536 sub-tests and keeps 3,257 after filtering. IntentTester generates 2,776 syntactically correct tests with 85% correctness, compared with 51% for MUT and 43% for METALLICUS. The generated tests expose 25 real defects, including stack overflow and null dereference bugs.

The corpus also includes early work on how teams maintain coding-agent instructions. The Agent Context File study examines files such as CLAUDE.md, AGENTS.md, and copilot-instructions.md as repository artifacts with commit histories. It plans to classify changes, connect them to later agent-generated code quality, and measure timing across development windows.

This paper reports study design and feasibility, not completed results. The scale still matters for grounding the question: the authors cite AIDev with 116,211 repositories and 932,791 pull requests involving agent-generated code, plus 2,303 context files from 1,925 repositories. A preliminary pipeline produced 10,763 context-file snapshots and 8,600 commits with overlapping instruction-file and agent-code information.