Coding-agent research is measuring user burden, runtime cost, and tool risk

Two benchmarks put the user back into software-engineering evaluation. SWE-Together rebuilds 109 repository tasks from real user-agent sessions and scores both final code quality and User Correction, a measure of explicit corrections and softer nudges. Claude Opus 4.8 leads its reported runs at 63% pass@1, while the reference patch baseline reaches about 78%.

SWE-INTERACT shows how much harder vague requests and delayed requirements are. On the same underlying tasks, Opus 4.8 drops from 50.7% single-turn resolve to 26.7% in the interactive setting. GPT-5.5 drops from 48.0% to 24.7%, while its per-trial cost rises from $2.78 to $9.84. The failure labels matter: many agents discover most goals, then still lose requirements or introduce implementation bugs.

MirrorCode tests whether agents can rebuild command-line programs from behavior alone. The setup gives execute-only access to a target program, documentation, and tests, then checks exact stdout and stderr matches. Claude Opus 4.7 averages a 56% perfect-solve rate across 25 targets, and one run reimplements gotree, a roughly 16,000-line Go bioinformatics toolkit, with 2,000 of 2,001 tests passing after 14 hours and $251.

TraceLab supplies the systems side of that ambition. Its trace covers 4,265 Claude Code and Codex sessions with 357,161 large language model (LLM) steps and 432,510 tool calls. The median step reads about 119K prefix tokens and writes 214 output tokens. Prefix reads account for 59.5% of estimated API cost, so cache policy and context reuse are central operating concerns for coding-agent products.

Agent safety work focuses on what happens after a model can call tools. The jailbreak essay argues that alignment changes output probabilities without creating hard execution rules, then connects prompt injection to ReAct-style agents where untrusted content and control instructions share a context window. Its examples include tool-using systems that can edit files, run shell commands, or act through account workflows.

trajeckt is a concrete runtime answer for Model Context Protocol (MCP) agents. It installs a sealed commitment graph before execution, checks each tool call against allowed order and data-flow rules, and blocks missing commitments by default. Its smoke test allows read_database and summarize, then blocks send_email_external with HTTP 403 when sensitive data would reach an external sink. A separate MCP pattern study adds design guidance: static tool aggregation can reduce tool-selection accuracy once visible tools grow past roughly 10–15 for Claude Haiku 4.5 and 20–30 for Claude Sonnet 4.

Several papers narrow the task and measure where extra agent structure helps. In README generation with retrieval-augmented generation (RAG), a single-agent system slightly beats an autonomous multi-agent system on ROUGE-L F1 while using about one seventh of the tokens. Human-written plans produce the best judged documentation, but they also cost more time and tokens.

Education papers add process-level measurements. Clover logs how students accept, ignore, edit, and delete AI code-completion suggestions, then inserts bad suggestions as attention checks. In a 55-student CS1 study, tab-acceptance rate strongly correlates with failed attention checks. PyMETA adds a separate diagnostic benchmark: 48,646 Python submissions with interpreter-derived labels and a 14-class error taxonomy. Together, these items treat code assistance as a behavior to audit, not just an output to score.