Source note
DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection
Summary
DREA improves repository-level vulnerability detection by combining a reasoning-focused Planner with a lightweight Explorer that gathers security-relevant code context on demand. On the 100-pair RepoPairBench benchmark, it raises pair-level correctness across three LLM backbones while shifting most exploration tokens to a local model.
Problem
- Function-only vulnerability detectors and fixed retrieval rules often miss cross-file data flows, object-scoped authorization, sanitization, and configuration details.
- This matters because a correct security judgment may depend on repository evidence outside the target function, while exhaustive exploration through an expensive model can be impractical.
Approach
- A stronger Planner forms a vulnerability hypothesis, requests targeted repository evidence, updates its hypothesis, and produces the final vulnerable or benign decision.
- A lightweight local Explorer uses read-only
ls,glob,grep, andread_filetools to retrieve repository context, code evidence, and security findings. - DREA evaluates 100 validated Python vulnerability-fix pairs from 2021–2025, covering 48 CWE categories, with both vulnerable and patched repository snapshots.
- The study adds reasoning-correctness evaluation using GPT-4.1 as an LLM judge, distinguishing sound true positives from correct labels supported by inaccurate rationales.
Results
- On RepoPairBench, DREA raises Pair-Correctness from 19% to 42% for DeepSeek-V3.2, from 26% to 34% for GLM-4.7, and from 21% to 30% for GPT-5.2, compared with the Function-Only Baseline.
- For DeepSeek-V3.2, DREA improves recall from 39% to 80%, F1 from 45.6% to 71.1%, and Youden’s J from 7 to 35 percentage points; its false-positive rate increases from 32% to 45%.
- DREA offloads more than 93% of tokens to the local Explorer and reduces estimated billable API cost by 16–48 times.
- Across evaluated systems, 26–55% of true-positive predictions are Lucky Hits: the label is correct but the rationale does not match the documented vulnerability mechanism.
- The reasoning judge agrees with two security researchers at 96.0% and 94.0%, with Cohen’s kappa values of 0.92 and 0.88; inter-annotator agreement is 92.0% with kappa 0.84.
- The reported evaluation is limited to 100 recent Python vulnerability-fix pairs and uses an LLM-as-a-Judge protocol for reasoning quality, so generalization to other languages, vulnerability datasets, and independent human evaluation remains uncertain.