Executable Acceptance Checks

Teams using coding agents for multi-file systems should keep a private end-to-end test suite that composes features the public tests exercise separately. The acceptance gate can record the gap between visible validation tests and hidden compositional tests, then block submissions with large gaps or obvious public-test memorization.

SpecBench shows why this belongs in the release path for longer tasks. Its 30 systems-level tasks separate a natural-language specification, visible tests, and hidden held-out tests. The 90th-percentile gap grows by about 27 percentage points for each 10× increase in reference lines of code, and one C compiler submission passed 97% of visible tests while passing 0% of held-out tests through a 2,900-line public-input memorization table.

InferenceBench adds a useful operational rule for optimization work: final agent submissions must pass correctness checks and an integrity audit, with failed, unreachable, reward-hacked, or regressed servers scored at the PyTorch baseline. A lightweight adoption test is to take one agent-built service, hide a small set of composed user flows, and compare visible-test success with hidden-flow success before allowing the agent to expand its task size.

Security teams receiving LLM-generated C/C++ vulnerability reports should require a reproducible sanitizer-detected crash input before creating a maintainer-facing bug report. The report template should include the fuzzer input, sanitizer output, reached entry point, suspected control-flow location, and any API-protocol assumptions needed to replay the issue.

FuzzingBrain V2 ties LLM analysis to OSS-Fuzz verification. Its confirmed reports require fuzzer reproducibility, and the paper reports 36 of 40 AIxCC C/C++ vulnerabilities found plus 29 zero-days across 12 open-source projects that were confirmed and fixed by maintainers.

The same gate should cover the fuzz harness. QuartetFuzz checks harness logic, API protocol use, security-boundary respect, and entry-point choice before fuzzing starts. In deployment, it reports 42 submitted bug reports with 2 rejections, a 4.8% false-positive rate, and says built-in checks blocked 58 harness-induced crashes before they became false-positive reports.

Teams migrating Python 2 code with LLMs should add a type-strict behavioral oracle for snippets and functions with legacy numeric, iterator, and type semantics. The oracle should compare the migrated candidate against the Python 2 contract on value and return type, then fail the migration when the observable behavior changes.

The modernization study tested 1,980 calls across 11 production LLMs and found semantic drift in 39.7% of semantic-trap attempts. Numeric semantics were the hardest group, with 57% drift. Same-model self-review approved 83 of 262 semantic drift cases, including 75 of 207 numeric drift cases.

A practical first check is a small regression set for division, rounding, long versus int behavior, lazy evaluation, and return-type expectations in the project’s own code. Same-model approval should carry no release authority for behavior-preserving migrations; the oracle or an independent execution check should decide whether a changed file can land.