Coding agents are being tested against project rules, full workflows, and measurement failures

Code generation papers target the decisions around code. Context-Augmented Code Generation reports that adding Brief to Claude Code raises weighted decision compliance from 46% to 95% on 8 Next.js tasks with 41 decision points. The result is useful, but the setup also changes the workflow through specs, acceptance criteria, and mid-build guidance.

The industrial DSL study makes the same point at repository scale. BMW encodes an Xtext domain-specific language project as path-preserving JSON, then trains 7B code models to output complete multi-file updates. QLoRA fine-tuning reaches structural fidelity of 1.00 on 105 held-out examples, with developer review and generator execution used as practical checks.

Agent systems are doing better when they pass explicit artifacts between stages. Mono2Sls uses static analysis to extract endpoints, call edges, async hints, and schema candidates before four agents generate the architecture, Lambda code, AWS SAM template, and consistency fixes. On 6 Flask and Express applications, it reports 100% deployment success without manual fixes and 66.1% end-to-end correctness.

FGDM applies a related staged design to bug detection and repair. It turns code into flow graphs, localizes faulty nodes, repairs graph regions, and reconstructs source. The paper reports tests on 100 BugsInPy programs converted across Python and C, with mean cosine similarity of 0.951 for Python repairs and 0.974 for C repairs against reference representations.

Evaluation work is examining the measurement tools themselves. BenchGuard audits execution-based agent benchmarks by checking task instructions, reference programs, evaluators, and environments for conflicts. On ScienceAgentBench it finds 12 author-confirmed defects across 102 tasks. On BIXBench Verified-50, a five-model audit matches 20 of 24 expert-identified atomic issues exactly.

TraceToChain takes a different route. It fits agent traces to an absorbing Markov chain so pass@k, repeated-run failure, and reliability decay share one success-time distribution. On 7 controlled setups, held-out reliability curves match analytic curves with maximum error 0.053, and the fitted chain passes the reported KS test in all 7 cases. Raw SWE-bench and tau-bench trace validation is still left for future work.

Two agent benchmarks test whether models can build systems after limited instruction or partial discovery. In the AlphaZero-style Connect Four task, agents get a short prompt, 3 hours, and consumer hardware. Claude Opus 4.7 wins as first mover against the Pascal Pons solver in 7 of 8 main trials, while other tested agents stay at 2 wins or fewer.

SciCrafter tests discovery and application inside Minecraft redstone. The best baseline reaches 26.0% success across 25 parameterized circuit tasks. Oracle investigation hints and a scientist sub-agent raise top results to 64.0%, but the remaining gap is large enough to show that agents still struggle to identify and apply missing causal rules in this controlled setting.