Targeted evidence for agent-mediated software changes

Open-source maintainers can map each contribution risk class to an executable testing obligation in the repository manifest. For ordinary library changes, the agent would identify modified functions and public entry points, generate old-versus-new tests, and attach the observed behavioral differences plus union coverage. Compiler or toolchain changes could instead trigger region-level fuzzing that records the program structures and options needed to reach relevant coverage gaps. DiffTestGen exposed differences in 78.2% of 463 pull requests, while GapForge’s coverage-gap targeting found 12 compiler failures; the Agent Governance Manifest separately showed that explicit risk and evidence rules improved reviewers’ exact risk-label recovery.

The concrete change is to add fields such as affected execution targets, permitted behavioral changes, required test mode, coverage evidence, and unresolved gaps to the manifest, with maintainers retaining the acceptance decision. A low-cost check is to replay recent agent-authored pull requests through this policy and compare newly exposed behavior and review time with the project’s current generic CI evidence.

Security teams maintaining reusable coding-agent workflows can test prompts as versioned security controls. The workflow test would parse each prompt, delete or move one guard, condition, qualifier, or concept binding at a time, generate code for a fixed set of security-relevant tasks, and compare CodeQL findings with the unmodified prompt across supported models. This applies software mutation testing to the instructions themselves.

The need comes from two incomplete but complementary observations: parser-driven experiments found that fine-grained prompt constituents affect insecure-code likelihood, while the Loop Library demonstrates that engineering teams are already packaging checks and stopping conditions as reusable natural-language workflows but reports no aggregate evaluation. Because the supplied prompt study omits effect sizes and per-model vulnerability rates, teams should first run this test on their own prompt and model matrix; clauses whose mutation repeatedly changes CWE incidence become protected regression cases rather than style guidance.

ML privacy and compliance teams reviewing third-party or agent-generated training code need a gate that can detect covert dataset-property channels even when task accuracy remains unchanged. Mark imported training pipelines as a high-risk contribution in repository policy, then require an instrumented clean-room run on paired synthetic datasets that differ in one sensitive aggregate property. Reviewers would inspect candidate code for synthetic-sample insertion and branches over dataset statistics, then test whether fixed label-only probes can reliably distinguish which dataset property was used during training.

CPPIA reports 100% property-inference accuracy across four datasets, eight architectures, and 18 properties without reducing model accuracy, so ordinary accuracy and regression checks would not expose the reported attack. The Agent Governance Manifest supplies a mechanism for assigning the risk class, evidence obligation, and maintainer gate. This paired run is only a screening test—the paper excerpt lacks per-dataset metrics and query counts—but a reproducible label signal would give reviewers concrete grounds to reject or isolate the pipeline before private data is used.