Software agents look strongest when their work is executable and access-scoped

ReaComp gives the clearest efficiency result. It turns about 100 LLM reasoning traces per benchmark into reusable Python symbolic solvers. On PBEBench-Hard, the symbolic ensemble reaches 84.7% accuracy with no test-time LLM tokens, while Best-of-K reaches 68.4%. The hybrid also cuts reported token use by 78%.

The ARC-AGI-3 work applies the same discipline to interactive games. The coding agent writes a Python world model, checks it against observed transitions, plans inside it, and executes actions only while predictions keep matching. On 25 public games it fully solves 7, with large run-to-run variance and no private-set result yet.

UVMarvel extends the pattern to hardware verification. It builds Universal Verification Methodology (UVM) testbenches for subsystem-level RTL, then uses coverage reports and signal tracing to ask LLMs for new sequences. The paper reports 95.65% average code coverage across six subsystem benchmarks.

Slyp is a strong example of tool-specific security automation. It gives an agent binary exploration, COM inspection, and live debugging tools for Windows Component Object Model (COM) services. On 40 vulnerability cases it reaches 0.973 F1, verifies proof-of-concept code for 27 cases in its strongest setup, and finds 28 previously unknown production vulnerabilities later confirmed by MSRC.

The safety picture is broader than bug finding. Agents of Chaos tested six autonomous agents for two weeks in a live Discord setting with memory, email, shell access, and human interaction. The study reports 10 security vulnerabilities and 6 cases where agents kept suitable boundaries.

Enterprise retrieval adds a different control point. The OGX design tags chunks with tenant and access metadata, applies authorization before and during retrieval, and keeps tool execution and conversation state on the server. In its reported tests, ungated retrieval leaked cross-tenant data in 98–100% of probes; ABAC gating reduced leakage and authorization violations to 0%.

Two papers treat context as an engineering input that must be written down before generation starts. Mise en Place for Agentic Coding records domain knowledge, specifications, and task records before parallel agents write code. Its evidence is a single hackathon case, so the paper is useful as a process report, not as a controlled result.

The platform-service scaffolding paper gives a more concrete deployment test. A retrieval-augmented generation (RAG) system selects approved Backstage templates after a short clarification chat. In the reported setup it chose the ground-truth template in 10 of 10 runs. In a small comparison, only 2 of 7 Copilot users passed all deployment quality gates, while the template-selection system passed them all with far fewer prompts and tokens.

Repository mining adds one more view of context. Agents with bash and git access classify commits, reviews, code lines, and repositories with similar accuracy to fixed-context LLM calls, while avoiding context-window overflows across 4,943 valid classifications. The cost is higher per run, but it scales less with artifact size.

The meta-analysis is the cautionary anchor for coding-assistant claims. Across 23 studies and 27 effect sizes, generative AI assistance shows a moderate positive programming-productivity effect overall. The setting matters: lab studies show a larger effect, while enterprise and open-source settings show small, non-significant effects in the reported moderator analysis.

Learning evidence is weaker. The pooled learning effect is small and non-significant. Gains appear when students can use AI during assessment, while blocked-assessment results do not show a reliable benefit.

The refactoring-adoption study shows how developers actually use suggestions. In 169 GitHub refactoring commits linked to ChatGPT conversations, many committed changes are either close copies or partial selections from longer suggestions. Readability and maintainability are the most common goals, but one repository supplies most commits, so the dataset is imbalanced.