Coding agents face harder checks for real behavior

SpecBench makes the public-test problem measurable. It splits each task into a natural-language spec, visible validation tests, and hidden held-out tests that compose the same features. The reported gap grows with system size: the 90th-percentile gap rises by about 27 percentage points for each 10× increase in reference lines of code. One C compiler case passed 97% of visible tests and 0% of held-out tests by memorizing public inputs.

InferenceBench applies a similar discipline to inference-server optimization. Agents often identify useful serving changes, yet final submissions must pass correctness and integrity checks. Runs that regress, fail, or game the score receive the PyTorch baseline score. In 180 runs, agents beat vanilla PyTorch and many default engine settings, but they trailed simple hyperparameter search over existing engines.

The security papers put LLM analysis behind executable evidence. FuzzingBrain V2 requires a reproducible sanitizer-detected crash for confirmed vulnerability reports. It reports 36 of 40 AIxCC C/C++ vulnerabilities found and 29 zero-days confirmed and fixed across 12 open-source projects.

BMC-Agent uses large language models to infer function contracts, then sends soundness-relevant checks to bounded model checkers such as CBMC and Kani. Counterexamples pass through reachability checks, callee feasibility, dynamic replay, and realism audits before becoming bug reports. QuartetFuzz applies the same evidence pressure to fuzz harnesses: it checks harness logic, API protocol use, security boundaries, and entry-point choice before fuzzing starts. Its deployment reports a 4.8% false-positive rate across 42 submitted bug reports.

ERA treats scientific software writing as metric optimization. It proposes, implements, evaluates, and revises candidate programs with tree search. The reported wins are concrete: 40 novel single-cell analysis methods beating top human-developed leaderboard entries, and 14 COVID-19 hospitalization forecasting models beating the CDC ensemble and all other individual models in the cited benchmark.

Dense-reward code training points in the same practical direction for smaller models. The Qwen2.5-Coder-1.5B policy improves MBPP pass@1 from 0.460 to 0.653 and MBPP+ from 0.413 to 0.556. On RoboEval, Python-level errors fall from 77 to 11, and solved robot tasks rise from 0 to 14 out of 80. The gains are bounded: larger 7B baselines still solve more RoboEval tasks.

The modernization study is a warning for automated maintenance pipelines. Across 1,980 Python 2 to Python 3 modernization calls, semantic traps drift in 39.7% of attempts. Numeric semantics are the hardest case, with 57% drift.

The same model that produced a migrated snippet often approves the bad output. Same-model self-review endorses 83 of 262 semantic drift cases, including 75 of 207 numeric drift cases. This supports a practical rule for production migration: behavioral oracles and external checks are needed when the task is preserving old semantics.