Recoleta Item Note

Patch Validation in Automated Vulnerability Repair

automated-vulnerability-repairpatch-validationllm-for-codebenchmark-datasetsoftware-security

Summary

This paper points out that the patch validation methods commonly used in existing Automated Vulnerability Repair (AVR) research overestimate repair success rates because they ignore the new tests developers add alongside official patches. The authors propose using PoC[+] tests for stricter validation and construct the PVBench benchmark, containing 209 real vulnerability cases, to quantify this issue.

Problem

Existing AVR evaluations typically only check whether the original functional tests pass and whether the PoC no longer triggers the vulnerability; however, this may fail to verify whether a patch aligns with developer intent.
When submitting real patches, developers often add new tests. These tests not only prevent crashes, but also encode root cause location, correct fix strategy, program specifications, and style constraints.
Therefore, if these newly added tests are ignored, the “repair success rate” of AVR systems will be systematically overestimated, which can mislead judgments about LLM repair capability.

Approach

The authors propose PoC[+] tests: tests newly added by developers around the official patch, using them to replace or supplement the traditional validation method of “original test suite + PoC.”
They build PVBench: covering 20 open-source projects, 209 vulnerability cases, and 12 CWE categories, with each case including both basic tests (original functional tests + PoC) and PoC[+] tests.
They re-evaluate 3 SOTA AVR systems: PatchAgent, San2Patch, SWE-Agent, comparing the gap between “judged correct by basic tests” and “judged correct by PoC[+].”
The authors also categorize PoC[+] tests into three types by mechanism: output-checking, intermediate-checking, self-checking, illustrating how these tests capture richer program semantics.
They manually analyze patches that pass/fail PoC[+], concluding that the main issues concentrate in root-cause analysis, program specification adherence, developer intention capture.

Results

The authors release PVBench, containing 209 real vulnerability cases from 20 projects and covering 12 CWE categories; examples include CWE-476 with 52 cases, CWE-617 with 40, CWE-122 with 34, CWE-416 with 32, and CWE-190 with 26.
Re-evaluation on PatchAgent, San2Patch, SWE-Agent finds that more than 40% of patches judged correct by traditional basic tests fail under PoC[+] testing, showing substantial false discovery / success-rate overestimation in existing evaluation methods.
To validate the reliability of PoC[+], the authors manually compare patches that pass PoC[+] with developer patches and find that more than 70% achieve semantic equivalence; the remainder mainly have performance or quality issues, indicating that PoC[+] captures developer intent reasonably well, though it is still not perfect.
The strongest concrete evidence on dataset size and distribution includes: php with 43 vulnerabilities, cpython with 33, llvm with 26, v8 with 24, libxml2 with 19, and icu with 15.
The paper excerpt does not provide precise per-tool success-rate numbers for the three AVR systems, nor a complete table broken down by dataset/tool; the most important quantitative conclusion is that >40% of patches deemed “correct” by basic tests are overturned by PoC[+].

Link

http://arxiv.org/abs/2603.06858v1

Built with Recoleta

Run your own research radar

Turn arXiv, Hacker News, OpenReview, Hugging Face Daily Papers, and RSS into local Markdown, Obsidian notes, Telegram digests, and a public site.

View repo 5-minute quickstart