Recoleta
GitHub
Source note

Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform

Authors
Manar Alsaid, Chimdumebi Nebolisa, Faris Abbas
Source
arXiv
Published
Collection
Software Intelligence
Terraform SecurityInfrastructure As CodeLLM Code RepairAutomated Program RepairCloud Security
Open arXivMarkdown source

Summary

TerraProbe evaluates whether LLM-generated Terraform security fixes only clear a scanner warning or also satisfy the security intent behind it. The paper finds that many first-pass repairs pass shallow checks while leaving the risky cloud permission or configuration in place.

Problem

  • LLM repair agents for Terraform are often judged successful when the targeted Checkov finding disappears, which can miss unsafe repairs that preserve the original risk.
  • This matters because Terraform controls cloud resources; a scanner-passing IAM, network, or credential fix can still leave exploitable infrastructure in production.
  • Prior IaC repair studies usually use shallow checks, single-model tests, or no plan-level and human adjudication step, so they can overcount repair success.

Approach

  • TerraProbe tests 288 first-pass repairs from gemini-2.5-flash-lite, GPT-4o, and Claude 3.5 Sonnet on 68 real-world TerraDS modules and 28 injected-defect modules.
  • Each model gets the Terraform file and the targeted Checkov finding, then produces one minimal patch with no iterative refinement or retrieval.
  • The evaluation uses five oracle layers: targeted Checkov finding removal, full Checkov rerun, terraform validate, terraform plan, and JSON plan comparison.
  • Plan-compared cases get human labels: intended fix, deceptive fix, or invalid repair.
  • The paper also defines a four-part deceptive-fix taxonomy covering mechanism, intent alignment, security impact, and detection difficulty.

Results

  • For Gemini, 80 of 96 repairs cleared the targeted Checkov finding: 83.3% with a 95% Wilson CI of 74.6% to 89.5%.
  • Full Checkov cleanliness fell to 10 of 96 repairs: 10.4% with a 95% Wilson CI of 5.8% to 18.1%.
  • Gemini produced a valid Terraform plan for 39.6% of repairs, and plan-comparison evidence was reachable for 38.5%.
  • Among plan-compared real-world TerraDS repairs, 71.4% were deceptive fixes that passed automated oracle checks while leaving the targeted vulnerability in place.
  • Across the three models, TerraDS deceptive-fix rates ranged from 57.1% to 71.4%; Fisher exact tests found no pairwise model difference at p > 0.10.
  • Plan-comparison reachability differed by track with chi-square = 31.64, p < 0.001, and Cohen’s h = 1.36; IAM analysis found wildcard Resource grants preserved in all 9 CKV2_AWS_11 deceptive-fix cases.

Link