---
source: arxiv
url: https://arxiv.org/abs/2605.03956v1
published_at: '2026-05-05T16:39:29'
authors:
- Shravya Kanchi
- Xiaoyan Zang
- Ying Zhang
- Danfeng Yao
- Na Meng
topics:
- code-intelligence
- software-security
- llm-agents
- test-generation
- supply-chain-security
- java
relevance_score: 0.78
run_id: materialize-outputs
language_code: en
---

# Generating Proof-of-Vulnerability Tests to Help Enhance the Security of Complex Software

## Summary
PoVSmith generates JUnit proof-of-vulnerability tests for Java applications that depend on vulnerable libraries. It uses a coding agent to find reachable vulnerable APIs, write tests, run them, repair failures, and ask an LLM to judge whether the test triggered the vulnerability.

## Problem
- Developers need executable PoV tests to decide whether a dependency CVE can affect their own application.
- Manual PoV test writing is hard because the exploit must pass through application code, library APIs, build settings, and runtime constraints.
- Existing tools for this task fail in most cases, according to the paper's comparison to prior work.

## Approach
- PoVSmith starts with a vulnerable library version and an application that calls the library.
- It prompts Codex to find public application methods that directly or indirectly call vulnerable library APIs, then records the call paths.
- For each call path, it prompts Codex with the vulnerable API, CVE or issue ID, affected library version, source method, call path, and an exemplar library-level exploit test.
- Codex writes and revises a JUnit test after build and execution feedback, with up to 5 repair iterations.
- A separate GPT-based evaluator reads the generated test plus build and execution logs to decide whether the test demonstrated the vulnerability.

## Results
- The evaluation uses 33 Java App-Lib pairs, covering 20 vulnerability entries across 29 Maven projects, 3 Gradle projects, and 1 plain Java project.
- PoVSmith found 158 unique application-level public entry points that call vulnerable library APIs; 152 were correct, for 96% accuracy.
- It generated 152 tests from the correct call paths; 141 compiled successfully.
- 84 of the 152 generated tests triggered vulnerabilities, a 55% success rate.
- The paper says PoVSmith outperformed the prior LLM-based method by producing many more vulnerability-triggering tests, but the excerpt does not give the baseline count.
- The authors filed 6 pull requests and 6 CVE requests based on generated tests that demonstrated previously unreported vulnerabilities.

## Link
- [https://arxiv.org/abs/2605.03956v1](https://arxiv.org/abs/2605.03956v1)