Source note
DeepGuard: Secure Code Generation via Multi-Layer Semantic Aggregation
Summary
DeepGuard hardens code LLMs against insecure code generation by using security signals from multiple transformer layers instead of only the final layer. It combines multi-layer feature aggregation, security-aware fine-tuning, and a cheap inference-time logit bias to raise secure-and-correct code generation rates.
Problem
- Code LLMs often reproduce insecure coding patterns from training data, and prior work reports large failure rates such as about 40% vulnerable code in Copilot-generated samples.
- Many security-tuning methods supervise only the final transformer layer, but the paper shows vulnerability cues peak in intermediate-to-upper layers and weaken near the output layer.
- This matters because secure code generation must improve security without breaking functional correctness, which is the main value of code models in development workflows.
Approach
- DeepGuard probes model layers and finds that vulnerability-discriminative information is strongest in a band of upper layers, not at the last layer alone.
- It builds an attention-based aggregator over the top N layers to fuse their hidden states into a single security-sensitive representation for each token.
- A small security analyzer scores tokens and sequences using the aggregated representation plus a learned token-level security embedding; training pushes secure code to score higher than matched vulnerable code with a margin loss.
- The model is adapted with LoRA under a multi-objective loss: next-token loss on secure code, a security contrastive loss between vulnerable and secure pairs, and KL regularization to stay close to the base model.
- At inference, DeepGuard computes one prompt-conditioned security score, combines it with a token prior learned from secure vs. vulnerable training data, and adds a fixed bias to logits during decoding without per-step analyzer calls.
Results
- Across five code LLMs, DeepGuard improves the secure-and-correct generation rate by 11.9% on average over the strong baseline SVEN.
- On Qwen2.5-Coder-3B, sec-pass@1 rises from 70.47% with SVEN to 80.76% with DeepGuard; pass@1 is 86.65%, which the paper says is close to the original model.
- On Qwen2.5-Coder-7B, DeepGuard reports pass@1 83.18%, sec@1_pass 88.19%, sec-pass@1 73.35%, and SVEN-SR 89.21%.
- On DeepSeek-Coder-1.3B, DeepGuard reports pass@1 81.06%, sec@1_pass 84.91%, sec-pass@1 68.82%, and SVEN-SR 87.71%.
- On DeepSeek-Coder-6.7B, DeepGuard reports pass@1 88.47%, sec@1_pass 79.52%, sec-pass@1 70.35%, and SVEN-SR 81.82%.
- On Seed-Coder-8B, DeepGuard reports pass@1 86.59%, sec@1_pass 93.21%, sec-pass@1 80.71%, and SVEN-SR 93.21%; the paper also claims generalization to held-out vulnerability types, but the excerpt does not provide the held-out split numbers.