Code agents are being tested as bounded workers, not code generators

New benchmarks treat agents as workers that must plan, edit, run tools, and recover from mistakes. 1GC-7RC gives agents seven machine-learning tasks on one NVIDIA A100 GPU, with no internet access and 40–120 minute task budgets. AgentKernelArena tests 196 GPU kernel tasks across HIP, Triton, and PyTorch-to-HIP paths, then checks compilation, correctness, speed, and hidden input shapes. TOBench adds multimodal tool use: 100 executable tasks, 27 Model Context Protocol servers, and task-specific verifiers.

The results are mixed in a useful way. Claude Code with Sonnet 4.6 improves over all seven visible 1GC-7RC baselines. AgentKernelArena reports high compilation and correctness on many kernel categories, with mean speedups up to 6.89×, but PyTorch-to-HIP kernels often fail on unseen shapes. TOBench is much harsher: the best model reaches 41.0% task success against a 94.0% human benchmark.

Several papers focus on deciding when code generation is trustworthy enough to use. CodeRefuser samples programs, runs generated tests, clusters outputs, and refuses prompts that look likely to fail. Its calibration target is task-level risk, with the default setting checking whether all three sampled solutions are wrong.

Training-time feedback is also getting more careful. The diffusion-code reinforcement learning study finds that Pylint-style static analysis can be a better reward than unit-test pass rate when most generated programs fail. On DiffuCoder, static checking raises HumanEval from 53.9 to 67.1 and reduces rollout time from 29.3 seconds to 26.5 seconds. Moderate hints help, but high hint ratios can hurt, so the useful signal depends on task difficulty and reward type.

Agent safety work in this period centers on traceable action, not only safer final answers. The provenance paper argues that tool-using agents need records that show causal contribution, execution traceability, and possible intervention points across design, deployment, and monitoring. Its evidence base is mostly conceptual and cited risk data, but it names a concrete gap: multi-step agent harm can involve developers, tool authors, platform operators, and deployers.

Package hallucination gives the risk a software-supply-chain example. A replication across Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.4-mini, Gemini 2.5 Pro, and DeepSeek V3.2 finds overall hallucination rates clustered between 4.62% and 6.10%. The most actionable finding is a shared set of 127 nonexistent package names hallucinated by all five models, including 109 PyPI names and 18 npm names.

The practical constraint is no longer only model quality. Token use is becoming a budget line for engineering groups. The Pragmatic Engineer report cites two companies with about 10× token-spend growth in six months, one seed-stage infrastructure company rising from about $200 to about $3,000 per developer per month, and a SaaS company cutting costs 30% by changing the default model.

Tool design is responding at the code-editing layer. ane is an early terminal editor that lets agents read or edit a function, definition, line, or delimiter scope through Language Server Protocol support and headless CLI commands. The project has no benchmark evidence yet, but its design target is clear: reduce full-file reads and broad patches by giving agents narrower code operations.