Recoleta
GitHub
Source note

MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents

Authors
Jonathan Steinberg, Oren Gal
Source
arXiv
Published
Collection
Software Intelligence
Coding AgentsCode SecurityAgent BenchmarksSoftware VulnerabilitiesAI Code Review
Open arXivMarkdown source

Summary

MOSAIC-Bench shows that production coding agents can turn harmless-looking engineering tickets into exploitable software when a vulnerability is split across a normal three-ticket workflow. The benchmark uses runnable proof-of-concept oracles, so exploitability is checked on deployed test applications rather than judged by a model.

Problem

  • Current coding-agent safety tests often use one direct prompt, which misses vulnerabilities that appear only after several routine tickets are implemented in sequence.
  • This matters for Jira-style development workflows: a malicious ticket author with backlog access can avoid explicit harmful wording while still steering an agent toward exploitable code.
  • Standard AI code reviewers also miss part of the risk, approving confirmed-vulnerable cumulative diffs as normal pull requests.

Approach

  • The paper introduces MOSAIC-Bench: 199 three-stage attack chains across 10 web-application substrates, 31 CWE classes, and 5 programming languages.
  • Each chain contains three Jira-style tickets, a composed implementation, metadata, and a deterministic Python proof-of-concept oracle that returns VULNERABLE or SECURE against a Docker deployment.
  • The authors test nine production coding agents from Anthropic, OpenAI, Google, Moonshot, Zhipu, and Minimax under a fresh-session-per-ticket protocol.
  • They compare staged tickets with matched direct prompts, then test reviewer agents under diff-only, full-context, neutral, and pentester-framed review protocols.

Results

  • Across nine coding agents, staged tickets produced end-to-end attack success rates of 53.3% to 85.9%; GPT 5.3 reached 85.9%, and Claude Opus 4.6 reached 53.3%.
  • The staged runs had only 2 refusals across the full benchmark, while matched direct prompts over four Claude/Codex agents produced vulnerable-output rates of 0% to 20.4%.
  • At least one of the four frontier coding agents exploited 95% of chains.
  • Neutral code reviewers approved 24.8% of oracle-confirmed vulnerable cumulative diffs as routine PRs; full-codebase context reduced evasion to 16.4% on average.
  • Retaining conversation memory across all three stages closed only about 50% of the staged/direct gap at the median; GPT 5.4 still had 59.8% ASR with full memory.
  • Pentester framing reduced reviewer evasion to 3.0% to 17.6% on the evaluated reviewer subset; Gemma-4-E4B-it detected 88.4% of attacks with a 4.6% false-positive rate on 608 real GitHub PRs.

Link