AI coding agents are being judged by controls, traces, and full-task cost

Several papers put the inspection layer around AI coding work under pressure. The specification-governance paper ties AI coding gains to review load, context limits, and testable specs. It cites positive task-level studies, then also reports evidence where experienced developers slowed down on mature codebases and delivery stability fell with higher AI adoption.

AutoMat gives the hardest empirical warning. The best tested coding-agent setting reproduced 54.1% of 85 computational materials-science claims. Paper-only reproduction had near-zero success across systems, which points to missing procedures, domain tools, and fragile execution as current failure points.

RECAP attacks the measurement problem inside the editor. It records Copilot chat, shadow git commits, and fine-grained edits in VS Code. In one course deployment, it captured 2,034 prompts and 8,239 commits, making repeated error loops and AI edit share visible at session level.

The reward story is cautious. The pass-rate reward study tests reinforcement learning (RL) for code generation with DeepSeek-R1-Distill-Qwen-7B, Qwen3-4B, and Qwen2.5-7B-Instruct. Dense partial-credit feedback did not improve final pass@k over binary pass-all-tests rewards. On Qwen3-4B with GRPO, binary reward beat pass-rate reward on average pass@1, 46.4% versus 44.2%, and on pass@16, 59.1% versus 56.8%.

Themis broadens what a code reward model should score. It builds a benchmark with about 8.9k pairwise preferences across eight languages and five criteria: correctness, runtime, memory, maintainability, and security. The paper’s numeric model gains are not in the excerpt, so the grounded contribution is the benchmark, training data, and multi-criteria setup.

SAGA shows that serving an agent is different from serving isolated LLM calls. Agent tasks often make 10 to 100 chained calls with tool gaps in between. SAGA schedules the whole workflow, preserves reusable key-value cache across tool calls, and routes later steps back to the same worker. On 64 A100 GPUs, it reduced task completion time by 1.73x on SWE-bench and 1.55x on WebArena versus vLLM with Automatic Prefix Caching.

A separate position paper makes a control-layer argument for Bayesian decision rules. It gives no experiments, so its value is conceptual: the orchestration layer can track task uncertainty, agent reliability, cost, and stopping decisions without making the language model itself Bayesian.