Postgeneration Verification Controls

Static-analysis chat should stop asking the model to write Joern CPGQL directly. The better build is a two-step translator: the model fills a small typed JSON form for query intent, then deterministic code maps that form into the final query and runs validation before execution. The evidence is unusually concrete. In a 20-task benchmark over Apache Commons Lang and OWASP WebGoat, the schema-bound JSON design beat direct query generation for every tested model, with gains of 15.0 points for Qwen 72B and 25.0 points for Llama 70B. It also beat an agentic tool loop that used about 8 times more tokens per task and still produced worse results.

The operational pain is familiar to teams building natural-language access for static analysis: generated queries often execute but answer the wrong question, while free-form agent loops add latency and failure modes without solving DSL correctness. A typed intermediate gives product teams a place to enforce allowed query types, required fields, and schema checks before anything touches the analysis engine.

A cheap validation path is straightforward. Take a backlog of real English analysis requests from security or AppSec users, define a narrow JSON schema for the top query families, and compare result-match rate, execution success, and token cost against direct query generation. If the tool already logs failed or misleading CPGQL generations, those requests are the right starting set.

Teams with decent test suites can add runtime semantic checking by compiling selected tests into always-on checkers for production and staging. FlyCatcher points to a concrete way to do this: use existing tests as the source of intended behavior, infer a checker with static analysis plus LLM synthesis, track the needed abstract state in shadow state, and validate the generated checker on held-out tests before deployment.

This fits a real gap in coding-agent workflows. Generated code may pass the local test suite and still violate semantics later under different workloads. Runtime checkers cover that gap for failures that do not crash the program or trip ordinary assertions. On 400 tests from four Java systems, FlyCatcher inferred 334 checkers, with 300 judged correct through cross-validation. It produced 2.6 times more correct checkers than T2C and detected 5.2 times more mutants.

The first users are teams running stateful Java services where silent failures are expensive: messaging, replication, session handling, inventory logic, or any code with invariants that matter after deployment. The cheap check is to pick one subsystem with high-value integration tests, generate checkers for a small batch of tests, and measure three things in staging: validation pass rate for the generated checker, extra mutant detection, and runtime overhead. The paper’s overhead range, 2.7% to 40.3%, makes that measurement necessary before broad rollout.

A useful next step for repository test bots is to start from one developer-written test and expand it into a family of scenario variants, then repair those tests until they compile and pass. TestGeneralizer gives a concrete recipe for this workflow. It infers the scenario behind the seed test, writes a scenario template with variation points, instantiates those variants, and uses project facts from program analysis to keep the generated tests valid.

The user pressure is clear in mature repositories: the first test often captures one happy path, while the rest of the scenario coverage arrives later through bug reports and patch releases. This paper targets that gap directly. Across 12 Java projects, 506 focal methods, and 1,637 test scenarios, TestGeneralizer improved mutation-based scenario coverage by 57.67% over EvoSuite, 37.44% over gpt-o4-mini, and 31.66% over ChatTester. The most persuasive adoption signal is the field study: 16 of 27 generated tests were accepted and merged by maintainers.

This is a buildable workflow for CI or for pull-request review on changed methods that already have at least one trusted test. A low-cost trial is to run it only on files with new or modified tests, limit output to two or three candidate variants per seed test, and track merge acceptance, flaky-test rate, and distinct bug findings. That setup keeps reviewer load bounded while showing whether scenario expansion is adding useful coverage or only more test volume.