Coding agents are being judged by operational evidence, review gates, and executable checks

Coding agents are now judged inside real developer workflows, not only on finished patches. One large study of 20,574 IDE and command-line sessions found 16,118 evidence-grounded misalignment episodes. Constraint violations were the largest symptom category at 38.33%, and only 9.33% of episodes showed visible resolution in the logs. Most visible fixes required explicit developer pushback.

RADAR shows the production side of the same issue. Meta routes low-risk diffs through source eligibility rules, a Diff Risk Score, large language model review, and deterministic checks before landing. The system reviewed more than 535K diffs and landed more than 331K, with lower revert and production-incident rates than non-RADAR diffs in the reported deployment.

Several papers treat correctness as a behavior claim that needs concrete checks. TRAILS targets the oracle problem for generated code by executing candidate programs on generated inputs, then asking a large language model to judge each input-output pair against the natural-language specification. On LiveCodeBench and CoCoClaNeL, it improves Matthew correlation coefficient over zero-shot chain-of-thought baselines, with higher token cost per task.

Projectional decoding adds checks during generation. It keeps a partial graph model beside the token stream and masks tokens that violate semantic constraints. On CLEVR domain-specific language programs, semantic validity reaches 73.33% to 79.67% across Qwen3 models. CODEFUSE-DeBench adds a caution for reverse engineering: recompilation alone is a weak proxy, since the best decompiler plus repair model reached 22.3% exact-plus-partial behavioral overlap and only 1.2% exact stdout match.

Security work in this period focuses on two practical failure modes: small prompt changes that create vulnerable code, and repair agents that forget useful fixes. The prompt-fragility study uses CWEval across five languages and finds that a single-character mutation can flip secure output to vulnerable output. Hidden-state probes reach about 0.70 mean held-out AUC for predicting the joint functional-secure target, with input-handling vulnerabilities easier to predict than secure-defaults flaws.

EvoRepair addresses repair repetition by storing scored repair experience across vulnerability attempts. Its loop retrieves related CVE or CWE experience, patches in Docker, summarizes the trajectory, and updates an experience bank. Using GPT-5-mini, it reports 93.47% on PATCHEVAL and 87.00% on SEC-bench, ahead of 12 automated vulnerability repair baselines in the paper’s comparison.

Agora shows that protocol bug finding needs agents with explicit state and domain constraints. The system splits work across an orchestrator, a strategy agent, and a test-generation agent. It creates hypotheses with trigger conditions, action sequences, expected faulty behavior, and oracle checks.

On Raft, EPaxos, HotStuff, and BullShark implementations, Agora reports 15 previously unknown protocol-level safety bugs. ReAct-style baselines using GPT-5.2, Gemini 3.0 Pro Preview, Claude Sonnet 4.5, and Qwen3 Coder found 22 implementation bugs in total, but no protocol-level logic bugs in the reported tests.