Recoleta
GitHub
Source note

The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs

Authors
Matteo Cicalese, Antonio Della Porta, Stefano Lambiase, Emanuele Iannone, Torge Hinrichs, Riccardo Scandariato, et al.
Source
arXiv
Published
Collection
Software Intelligence
Secure Code GenerationOpen LlmsPrompt EngineeringCode SecurityStatic Analysis
Open arXivMarkdown source

Summary

This paper studies whether fine-grained prompt syntax changes the security of code generated by open LLMs. It proposes parser-driven prompt perturbations to identify syntactic elements that may increase vulnerability risk.

Problem

  • LLM-generated code often contains security vulnerabilities, while prior prompt-engineering research mainly studies high-level instructions and proprietary models.
  • The paper asks whether removing specific clauses, guards, qualifiers, or other syntactic constituents changes vulnerability incidence, which matters for safer and more reproducible self-hosted code generation.

Approach

  • The authors parse 150 security-relevant prompts from LLMSecEval and create variants by removing exactly one syntactic constituent at a time.
  • They characterize each removal by constituent type, granularity, and sentence position, then analyze individual features and their combinations.
  • They generate code with Qwen 2.5 32B, Athene-V2 72B, and Phi-4 14.7B in C, Java, and Python.
  • They use CodeQL's default language query suites to label a snippet as vulnerable when at least one CWE is detected, applying chi-squared tests, Cramer's V, Barnard's exact test, risk ratios, and Benjamini-Hochberg correction.

Results

  • The experiment contains 150 baseline prompts and 4,320 prompt variants per language, for 12,960 prompts across C, Java, and Python.
  • The three open models produced 40,230 code solutions in total.
  • The excerpt reports that opening clauses, late-sentence removals, and omitted guards, qualifiers, or concept bindings are associated with higher vulnerability rates.
  • No exact vulnerability rates, statistical significance values, effect sizes, risk ratios, or model-by-language comparisons are provided in the supplied text because the empirical-results section is truncated.

Link