Recoleta
GitHub
Source note

Vulnerability Detection with Interprocedural Context in Multiple Languages: Assessing Effectiveness and Cost of Modern LLMs

Authors
Kevin Lira, Baldoino Fonseca, Davy Baía, Márcio Ribeiro, Wesley K. G. Assunção
Source
arXiv
Published
Collection
Software Intelligence
Vulnerability DetectionLLM EvaluationInterprocedural AnalysisSoftware SecurityMulti Language Code
Open arXivMarkdown source

Summary

This paper tests whether giving LLMs caller or callee code helps them find vulnerabilities that depend on cross-function behavior. Across 509 ReposVul vulnerabilities in C, C++, and Python, extra interprocedural context usually did not help and often hurt accuracy while increasing cost.

Problem

  • Prior LLM vulnerability studies often inspect one function at a time, which misses bugs that only appear when data or control flow crosses function boundaries.
  • Security teams also need results across multiple languages, plus usable explanations and reasonable inference cost for deployment in real tools.
  • The paper asks whether adding caller or callee context improves detection, how results change across C/C++/Python, and what cost and explanation quality each model delivers.

Approach

  • The authors evaluate four models: Claude Haiku 4.5, Gemini 3 Flash, GPT-4.1 Mini, and GPT-5 Mini.
  • They build an evaluation set of 509 interprocedural vulnerabilities from ReposVul, using AST analysis to identify caller-callee relations.
  • Each target function is tested under three prompt settings: code only (CO), code plus callees (CC), and code plus callers (CK).
  • They measure accuracy and F1, run paired McNemar tests with Holm-Bonferroni correction, estimate API cost from token counts, and manually rate 1,004 explanations for correctness and comprehensiveness on a 0-2 scale.

Results

  • Overall, code-only prompts were best for every model. Gemini 3 Flash scored 0.9853 accuracy / 0.9926 F1 in CO, Claude Haiku 4.5 scored 0.9756 / 0.9877, GPT-5 Mini scored 0.9072 / 0.9514, and GPT-4.1 Mini scored 0.7561 / 0.8611.
  • Adding context often reduced performance. GPT-4.1 Mini fell from 0.7561 accuracy in CO to 0.5401 in CC and 0.6923 in CK. GPT-5 Mini fell from 0.9072 in CO to 0.8348 in CC and 0.7742 in CK.
  • Statistical tests show significant context-related drops for some models: GPT-4.1 Mini CO vs CC had p=0.0000, Φ=0.404, GPT-4.1 Mini CC vs CK had p=0.0000, Φ=-0.436, and GPT-5 Mini CO vs CK had p=0.0003, Φ=0.259 after correction. Claude Haiku 4.5 showed no significant differences across context settings.
  • The abstract reports a language-specific cost result for C: Gemini 3 Flash achieved F1 ≥ 0.978 at an estimated $0.50-$0.58 per configuration, giving the best cost-effectiveness trade-off for C vulnerabilities.
  • Adding interprocedural context nearly doubled token use and inference cost without matching accuracy gains, according to the abstract.
  • For explanation quality, Claude Haiku 4.5 correctly identified and explained the vulnerability in 93.6% of evaluated cases. The abstract also states that the GPT models had zero-score rates 5× higher than Haiku 4.5 and Gemini 3 Flash for correctness and comprehensiveness.

Link