Executable Validation for Agent-Written Java Code

Teams that let coding agents work through ordinary backlog tickets should add a security check over the cumulative diff, not only the latest ticket. MOSAIC-Bench shows why: three harmless-looking tickets can compose into an exploitable application change, and nine production coding agents reached 53.3% to 85.9% attack success on staged chains with only two refusals across the benchmark.

A practical version is a CI job for agent-authored multi-ticket work: deploy the changed service in Docker, run deterministic exploit oracles for the affected surface, and require a second reviewer prompt that reads the full cumulative diff with an adversarial security role. MOSAIC-Bench found that neutral reviewer agents approved 24.8% of oracle-confirmed vulnerable cumulative diffs as routine pull requests, while pentester framing reduced evasion on the evaluated subset. The first adoption point is high-risk web application work where agents touch authentication, authorization, object mapping, deserialization, file handling, or query construction across several tickets.

Java teams triaging dependency alerts can ask a coding agent for a runnable JUnit proof-of-vulnerability test tied to an application entry point. PoVSmith gives a concrete recipe: find public application methods that reach vulnerable library APIs, generate a JUnit test with the vulnerable API, CVE or issue ID, affected version, call path, and an exemplar exploit test, then repair the test using build and execution logs.

This fits the security engineer’s recurring problem: a scanner says a library version is affected, but the team needs evidence that the vulnerable behavior is reachable through its own code. In PoVSmith’s evaluation across 33 Java application-library pairs, the system found 152 correct call paths, compiled 141 generated tests, and triggered vulnerabilities in 84 cases. A small internal trial could run this workflow on recent dependency alerts and measure three simple outcomes: correct call path, compiling test, and vulnerability-triggering execution.

Java maintainers can require an agent-generated reproduction test before work starts on a bug fix. The test has to fail on the current buggy code and pass after the fix, giving the reviewer an executable check that the issue was reproduced and then removed.

TDD-Bench-Java and e-Otter++ show the workflow in enough detail to copy for enterprise Java repositories: localize likely files and functions from the issue report, create one new Java test class in the right package and test directory, run it on the old code, read build or test logs, and revise for up to 10 iterations. On 250 Java issues from 13 open-source repositories, e-Otter++ reached 43.6% fail-to-pass with Claude-Sonnet-4.5 and 46.4% with GPT-5.2. Execution-based refinement added 9.4 and 13.6 percentage points over the initial generator, which points to build-log repair as the useful support layer.