Recoleta
GitHub
Source note

QASecClaw: A Multi-Agent LLM Approach for False Positive Reduction in Static Application Security Testing

Authors
Mohd Ruhul Ameen, Md Takrim Ul Alam, Akif Islam
Source
arXiv
Published
Collection
Software Intelligence
Code IntelligenceSastLLM AgentsSoftware SecurityFalse Positive Reduction
Open arXivMarkdown source

Summary

QASecClaw uses Semgrep to find candidate security issues, then uses a coding-focused LLM agent to suppress likely false positives after reading code context. On OWASP Benchmark v1.2, it reports much higher precision and F1 than standalone Semgrep, with a small recall drop.

Problem

  • SAST tools often flag safe code as vulnerable when they match a risky pattern but miss nearby context such as sanitization, encoding, validation, or parameterized APIs.
  • False positives waste developer and security-review time, reduce trust in scanner output, and can cause real vulnerabilities to be ignored.
  • The paper targets false positive reduction while keeping SAST as the high-recall source of candidate findings.

Approach

  • QASecClaw runs Semgrep first and treats its alerts as candidate vulnerabilities rather than asking an LLM to search the whole codebase.
  • A SAST Filter Agent sends each finding, CWE type, file location, and source-code context to Qwen 3.5 Plus and asks for a true-positive or false-positive judgment.
  • A Mission Orchestrator coordinates agents for test planning, Semgrep-based validation, evidence correlation, LLM filtering, and report generation.
  • The system processes findings in batches of 15 files and validates the LLM response as structured JSON.
  • If the LLM fails, times out, or returns malformed JSON, QASecClaw keeps the original Semgrep finding to avoid hiding a possible real issue.

Results

  • Evaluation uses the full OWASP Benchmark v1.2: 2,740 Java test cases across 11 CWE categories, with 1,415 vulnerable cases and 1,325 safe cases.
  • QASecClaw reports F1 = 90.93% versus 78.39% for standalone Semgrep, a gain of about 12.54 percentage points.
  • False positives fall from 560 with Semgrep to 64 with QASecClaw, an 88.6% reduction, while recall drops by 3.1%.
  • Aggregate metrics in the paper show QASecClaw at precision 0.951, recall 0.871, F1 0.909, false positive rate 0.048, and Youden’s J 0.823; Semgrep scores precision 0.695, recall 0.900, F1 0.784, false positive rate 0.423, and Youden’s J 0.477.
  • Per-CWE results cited in the abstract include SQL Injection F1 = 94.05%, Cross-Site Scripting F1 = 89.58%, and Weak Cryptography F1 = 99.61%.

Link