Recoleta
GitHub
Source note

Generating Proof-of-Vulnerability Tests to Help Enhance the Security of Complex Software

Authors
Shravya Kanchi, Xiaoyan Zang, Ying Zhang, Danfeng Yao, Na Meng
Source
arXiv
Published
Collection
Software Intelligence
Code IntelligenceSoftware SecurityLLM AgentsTest GenerationSupply Chain Security
Open arXivMarkdown source

Summary

PoVSmith generates JUnit proof-of-vulnerability tests for Java applications that depend on vulnerable libraries. It uses a coding agent to find reachable vulnerable APIs, write tests, run them, repair failures, and ask an LLM to judge whether the test triggered the vulnerability.

Problem

  • Developers need executable PoV tests to decide whether a dependency CVE can affect their own application.
  • Manual PoV test writing is hard because the exploit must pass through application code, library APIs, build settings, and runtime constraints.
  • Existing tools for this task fail in most cases, according to the paper's comparison to prior work.

Approach

  • PoVSmith starts with a vulnerable library version and an application that calls the library.
  • It prompts Codex to find public application methods that directly or indirectly call vulnerable library APIs, then records the call paths.
  • For each call path, it prompts Codex with the vulnerable API, CVE or issue ID, affected library version, source method, call path, and an exemplar library-level exploit test.
  • Codex writes and revises a JUnit test after build and execution feedback, with up to 5 repair iterations.
  • A separate GPT-based evaluator reads the generated test plus build and execution logs to decide whether the test demonstrated the vulnerability.

Results

  • The evaluation uses 33 Java App-Lib pairs, covering 20 vulnerability entries across 29 Maven projects, 3 Gradle projects, and 1 plain Java project.
  • PoVSmith found 158 unique application-level public entry points that call vulnerable library APIs; 152 were correct, for 96% accuracy.
  • It generated 152 tests from the correct call paths; 141 compiled successfully.
  • 84 of the 152 generated tests triggered vulnerabilities, a 55% success rate.
  • The paper says PoVSmith outperformed the prior LLM-based method by producing many more vulnerability-triggering tests, but the excerpt does not give the baseline count.
  • The authors filed 6 pull requests and 6 CVE requests based on generated tests that demonstrated previously unreported vulnerabilities.

Link