Source note
AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection
Summary
AnyPoC turns bug-report validation into executable proof-of-concept generation, so LLM bug finders can produce concrete evidence instead of text-only claims. It targets large, mixed software systems and aims to reject bad reports instead of generating plausible but fake PoCs.
Problem
- LLM bug detectors can propose many candidate bugs, but developers still need to manually verify whether each report is real.
- Naive LLM PoC generation is unreliable: models tend to produce outputs that look successful, including non-working PoCs or hallucinated execution evidence.
- Existing PoC generators are narrow in scope. They often depend on one language, one bug class, one scaffold, or datasets that contain only true bugs.
Approach
- AnyPoC is a multi-agent validator that takes a candidate bug report and either returns an executable PoC or rejects the report.
- An analyzer agent first fact-checks the report, checks code context and claimed bug mechanics, and filters some false reports before expensive generation starts.
- A generator agent then iteratively writes and runs a PoC, debugs it, and performs a second execution pass to collect explicit evidence such as logs and traces.
- A checker agent independently re-executes the PoC and verifies that the evidence actually demonstrates the bug, which is meant to reduce hallucinated success and reward hacking.
- A knowledge extractor and filter maintain a self-evolving knowledge base so later PoC attempts can reuse project-specific setup knowledge instead of rediscovering it each time.
Results
- Evaluated on 12 real-world software systems across different languages and domains, including Chromium, Firefox, LLVM, OpenSSL, SQLite, FFmpeg, and Redis.
- Compared with baseline coding agents such as Claude Code and Codex, AnyPoC produces 1.3x more valid PoCs for true-positive bug reports.
- It rejects 9.8x more false-positive bug reports than those baselines.
- In one cited failure case for a baseline, Claude Code with Opus 4.5 produced 142 plausible PoCs for 144 bug reports, but only 26 were valid.
- The paper claims 122 new bugs found to date, with 105 confirmed by developers, 86 fixed, and 45 generated PoCs adopted as official regression tests.
- Table 1 positions AnyPoC as broader than prior systems: language-agnostic, dependency-free, able to handle false-positive filtering, and credited with 122 new bugs versus smaller or narrower prior results.