Bounded LLM Review Gates

Security teams that already run Semgrep can add an LLM review step after the scanner, using the Semgrep finding, CWE label, file location, and nearby source context as the only input. The output should be a structured true-positive or false-positive judgment plus a short reason. If the model times out or returns malformed JSON, the workflow should keep the original Semgrep alert open.

QASecClaw gives this shape a useful test case. On OWASP Benchmark v1.2, it cut Semgrep false positives from 560 to 64 while recall dropped by 3.1%. That is enough evidence for a limited pilot on noisy rule families such as injection, XSS, and weak cryptography. The first local check should compare model-filtered alerts against recent human triage decisions and report both false-positive reduction and missed real findings.

Teams with many GitHub Actions workflows can turn documented CI practices into a fixed audit checklist and route only uncertain cases to humans. The useful unit is one workflow file checked against specific YES, NO, or NOT APPLICABLE questions, such as least-privilege permissions, pinned actions, secrets handling, caching, and failure notifications.

The GitHub Actions audit study shows why the queue needs disagreement handling. Four open-weight models had only fair agreement on 95 Java workflows, with Fleiss’ kappa at 0.28. The study accepted unanimous or near-unanimous answers, sent split cases to GPT-5, and used targeted human review for unresolved items. That process reduced verification effort by 81% while keeping 87% agreement with expert judgment. A practical rollout can start as a weekly report on permission controls, since the study found only 4% compliance there.

IDE vendors and Java teams can test an LLM review pass for risky refactorings by giving the model only the original code, the refactored code, and a correctness question. The check fits refactoring because the evidence is paired and concrete, and the model can return a short explanation for developer inspection.

The refactoring-oracle study built 226 real Java refactoring bug cases across 47 refactoring types, including compilation-error and behavior-change cases. GPT-5.4 reached 93.8% first-run accuracy, and GPT-OSS-20B reached 80.5%. A useful adoption test is a non-blocking warning on high-volume refactorings such as Move Method, Inline Method, Pull Up Method, Extract Local Variable, and Rename Method, compared against compiler results, existing tests, and developer accept-or-dismiss behavior.