LLMs gain credibility when software tasks give them narrow evidence

Security work has the clearest empirical signal. QASecClaw keeps Semgrep as the high-recall scanner, then asks a coding-focused LLM to judge each candidate finding with source context. On OWASP Benchmark v1.2, false positives fall from 560 to 64, while recall drops by 3.1%. That is a useful trade for teams that already drown in scanner noise.

VulKey applies a similar discipline to repair. It does not pass only a CWE label or a full example patch. It predicts a repair pattern with a syntactic action and a security-specific key element, then conditions patch generation on that pattern. On PrimeVul, it reports 31.5% repair accuracy, 7.6 percentage points above the best baseline cited in the summary. The result supports a practical lesson: security models need compact, task-specific evidence, not just vulnerability names.

Refactoring correctness looks like a strong fit for model-based triage because the input is paired and concrete: original Java code, refactored Java code, and a correctness question. The refactoring-oracle study builds 226 real IDE bug cases across 47 refactoring types. GPT-5.4 reaches 93.8% first-run accuracy, and GPT-OSS-20B reaches 80.5%. The authors also test stability under semantics-preserving code changes.

Commit classification is weaker. The Conventional Commit study uses 3,200 InfluxDB commits and prompt-only open models. The best single run reaches 0.6154 accuracy, and few-shot prompting is the best average prompt style. The result is still useful, but it reads as prompt-selection guidance for release automation rather than a production-grade classifier.

Agent work is paying closer attention to what the model actually sees at each step. ACDL gives researchers a way to describe the context window as role messages, information pieces, time-indexed state, conditions, loops, and multi-agent contexts. Its value is reproducibility and comparison: two ReAct-style agents can differ because one keeps reasoning traces in action history and another does not.

Conclave is a practical counterpart. It adds structured multi-model debate to OpenCode with LEAD, SUPPORT, ALIGN, BUILD, and CHALLENGE signals, then selects a winner by endorsement score. The project has no benchmark evidence yet, and its own cost accounting is clear: three models debating for three rounds make nine API calls per user message. The research question is therefore measurable output gain per added call.

The reliability discussion is becoming more concrete about what humans should inspect. The Knowledge Objects position paper proposes structured records for claims, procedures, evidence, scope, validation status, and provenance. It gives no new benchmark or user study, so its contribution is a design argument.

The GitHub Actions audit paper supplies an applied example of the same need. Four open-weight LLMs assess 95 Java workflows against a 30-item checklist, but inter-model agreement is only fair, with Fleiss’ kappa at 0.28. GPT-5 adjudication plus targeted human review reduces verification effort by 81% while keeping 87% agreement with expert judgment. That is a useful pattern for high-risk audits: model votes can reduce review load, while expert checks remain part of the process.