---
source: arxiv
url: https://arxiv.org/abs/2607.15937v1
published_at: '2026-07-17T13:20:21'
authors:
- Matteo Cicalese
- Antonio Della Porta
- Stefano Lambiase
- Emanuele Iannone
- Torge Hinrichs
- Riccardo Scandariato
- Fabio Palomba
topics:
- secure-code-generation
- open-llms
- prompt-engineering
- code-security
- static-analysis
relevance_score: 0.88
run_id: materialize-outputs
language_code: en
---

# The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs

## Summary
This paper studies whether fine-grained prompt syntax changes the security of code generated by open LLMs. It proposes parser-driven prompt perturbations to identify syntactic elements that may increase vulnerability risk.

## Problem
- LLM-generated code often contains security vulnerabilities, while prior prompt-engineering research mainly studies high-level instructions and proprietary models.
- The paper asks whether removing specific clauses, guards, qualifiers, or other syntactic constituents changes vulnerability incidence, which matters for safer and more reproducible self-hosted code generation.

## Approach
- The authors parse 150 security-relevant prompts from LLMSecEval and create variants by removing exactly one syntactic constituent at a time.
- They characterize each removal by constituent type, granularity, and sentence position, then analyze individual features and their combinations.
- They generate code with Qwen 2.5 32B, Athene-V2 72B, and Phi-4 14.7B in C, Java, and Python.
- They use CodeQL's default language query suites to label a snippet as vulnerable when at least one CWE is detected, applying chi-squared tests, Cramer's V, Barnard's exact test, risk ratios, and Benjamini-Hochberg correction.

## Results
- The experiment contains 150 baseline prompts and 4,320 prompt variants per language, for 12,960 prompts across C, Java, and Python.
- The three open models produced 40,230 code solutions in total.
- The excerpt reports that opening clauses, late-sentence removals, and omitted guards, qualifiers, or concept bindings are associated with higher vulnerability rates.
- No exact vulnerability rates, statistical significance values, effect sizes, risk ratios, or model-by-language comparisons are provided in the supplied text because the empirical-results section is truncated.

## Link
- [https://arxiv.org/abs/2607.15937v1](https://arxiv.org/abs/2607.15937v1)