Coding agents face a harder trust test: execution traces, repository discipline, and pre-action checks

Execution is the main filter for code-generation claims. Semantic Voting shows that running candidates on generated inputs gives a much stronger selection signal than text-level voting: execution-based selectors beat output-pattern majority voting by 19 to 52 percentage points across 18 HumanEval+ and MBPP+ settings. Sketch-and-Verify makes a related point for weaker code models. It asks the model to try distinct algorithmic sketches, then checks candidates by execution fingerprints. On 19 hard HumanEval+ cases for Gemini 3.1 Flash Lite, the smaller sketch setting solved 11 of 19 problems, compared with 5 of 19 for flat sampling at the same candidate count.

The same execution standard appears in testing work. ConCovUp targets concurrent C/C++ behavior that ordinary unit tests miss. It combines static shared-memory analysis, LLM-guided path reasoning, and run feedback, raising average shared-memory access pair coverage from 36.6% with a Claude Code baseline to 68.1% on nine real libraries.

Repository work is being measured beyond issue fixing. SWE Atlas covers 284 tasks across codebase Q&A, test writing, and refactoring in 18 active open-source repositories. Its evaluations combine program checks with rubrics for coverage, maintainability, conventions, cleanup, and code hygiene. The reported gap between regression-test pass rates and rubric pass rates in refactoring, about 15 to 40 points, shows why passing tests alone is an incomplete signal for engineering quality.

FixedBench adds a sharper maintenance requirement: the agent must know when to make no executable-code change. On already-fixed SWE-bench Verified issues, agents still made unwanted executable edits in 35% to 65% of cases under the main setting. Prompts that rewarded abstention improved no-change behavior, but caused harmful over-abstention when a partial fix still needed work.

Formal verification remains a hard boundary for current models. VeriContest contains 946 Rust/Verus competitive-programming tasks with natural-language prompts, expert specifications, accepted Rust code, checked proofs, and positive and negative tests. GPT-5.5 reaches 92.18% pass@1 on natural-language-to-code generation, yet only 5.29% end-to-end verified generation. All evaluated models stay below 6% end-to-end.

SmartEval covers another high-stakes domain: Solidity smart contracts generated from natural-language specifications. The benchmark evaluates 9,000 generated contracts against a five-metric rubric for functional completeness, variable fidelity, state-machine correctness, business logic, and code quality. The weakest metric is business-logic fidelity, and lower-performing contracts are dominated by logic omissions and state-transition errors.

Tool-using agents need checks before their first live action. RubricRefine inspects code-mode tool calls against task-specific contracts for tool choice, output shape, call signatures, ordering, and data provenance. On M3ToolEval it averages 0.86 success across seven models, compared with 0.62 for single-pass CodeAct, without executing the candidate action during refinement.

Security evaluation also gets more concrete. MOSAIC-Bench shows that ordinary-looking staged tickets can compose into exploitable code. Across nine production coding agents, staged attack chains reached 53.3% to 85.9% end-to-end attack success with only two refusals. MonitoringBench tests monitors with 2,644 successful BashArena attack trajectories and shows that stronger red-teaming can cut apparent monitor catch rates sharply, including a drop from 94.9% to 60.3% for an Opus 4.5 full-trajectory monitor on refined attacks.