Recoleta
GitHub
Source note

A Ground-Truth-Based Evaluation of Vulnerability Detection Across Multiple Ecosystems

Authors
Peter Mandl, Paul Mandl, Martin Häusl, Maximilian Auch
Source
arXiv
Published
Collection
Software Intelligence
Vulnerability DetectionSoftware Supply ChainGround Truth DatasetOsvSecurity Evaluation
Open arXivMarkdown source

Summary

This paper builds an OSV-based ground-truth dataset for package vulnerability detection and uses it to compare tools across npm, PyPI, Maven, and NuGet. Its main point is that tool results differ because data sources, identifiers, and version-range handling differ, so evaluations need a reproducible version-level reference.

Problem

  • Vulnerability scanners often disagree because vulnerability data comes from different databases with different identifiers, package mappings, and version-range formats.
  • Many advisories describe affected versions as ranges, and tools interpret those ranges with ecosystem-specific rules that can produce different answers for the same package version.
  • There is no stable, widely accepted ground truth that says which exact package versions are vulnerable, which makes tool evaluation hard to reproduce.

Approach

  • The authors derive a ground-truth dataset from a fixed snapshot of the Open Source Vulnerabilities (OSV) database for four ecosystems: npm, PyPI, Maven, and NuGet.
  • They normalize vulnerability identifiers and represent each fact as an explicit tuple (ecosystem, component, version, vulnerability) instead of leaving vulnerabilities as abstract version ranges.
  • They select widely used components, gather a bounded set of recent stable releases from each ecosystem registry, and exclude pre-release and development versions.
  • For each concrete package version, they query OSV to ask whether that exact version is affected; each returned vulnerability becomes one ground-truth entry.
  • They also provide an open-source reconstruction tool so the same dataset-building process can be rerun on newer OSV snapshots.

Results

  • The paper claims systematic differences across vulnerability detection systems when they are evaluated against the same OSV-derived ground truth.
  • It claims the dataset enables deterministic, reproducible comparison by matching tool outputs against explicit (e,c,v,u) tuples.
  • The study covers 4 ecosystems: npm, PyPI, Maven, and NuGet.
  • Example analyzed components include esbuild, vite, requests, keras, org.apache.logging.log4j, org.springframework:spring-expression, Microsoft.Data.SqlClient, and Microsoft.AspNetCore.Identity.
  • The excerpt does not include quantitative evaluation results such as precision, recall, false-positive counts, dataset size, or per-tool comparisons, so no numeric performance claims can be extracted from the provided text.

Link