Executable evidence dominates agent software reliability research

Several code-generation papers treat execution traces as the main signal for choosing or rejecting large language model (LLM) outputs. Semantic Voting clusters candidate programs by sandboxed behavior on generated inputs; across 18 HumanEval+ and MBPP+ settings, execution-based selectors beat output-pattern majority voting by 19 to 52 percentage points. SketchVerify adds structured candidate generation: it asks for different algorithmic sketches, fills them, then verifies candidates by execution fingerprints. On 19 hard HumanEval+ problems for Gemini 3.1 Flash Lite, K=2,M=5 solved 11 problems versus 5 for flat N=10 sampling.

Uncertainty work follows the same practical route. Semantic distance estimation (SDE) and directed SDE compare sampled programs by graded behavioral distance on fuzzed inputs. The directed variant reports AUROC 0.844 for GPT-4o-mini on LiveCodeBench pass@1 failure prediction, beating listed baselines such as DiffTrust and Semantic Entropy in the summary table.

Program repair work is giving learning systems finer credit for what changed. BoostAPR trains on execution-verified demonstrations, then uses a sequence reward model and a line-level reward model during PPO. On SWE-bench Verified, it reaches 40.7% pass@1 on 500 tasks, up from 17.8% for the Qwen2.5-Coder-32B base model. The line-level reward adds 2.4 points over sequence-only PPO in the reported result.

PROBE addresses a later stage of the same reliability problem: what to do after an agent fails. It records span-level telemetry, localizes the failure anchor, and emits bounded retry guidance only when the diagnosis is grounded and actionable. Across 257 initially unresolved cases, it reports 65.37% Top-1 diagnosis accuracy and 21.79% recovery. The gap between those two numbers is useful: finding the fault is easier than producing a next run that succeeds.

Agent reliability papers also put checks around the environment that agents touch. SkillGuard turns stale skill-library assumptions into contracts over operational items such as package versions, URLs, API paths, environment variables, and configuration files. On DriftBench, it reports 0 false positives across 599 no-drift and hard-negative cases, while contract-free CI probes produce 40% false positives. In a live scan of 49 real skills, one-round contract-guided repair reaches 78% success.

Formal verification appears in two forms. Containment verification proves boundary policies in the agent runtime, using Dafny to verify a PocketFlow action interface with allowed reads, tool calls, and step bounds. A separate compiler study gives cost data: full Lean 4 verification of coding-agent compiler optimizations takes about 7.6 to 19.9 times more active development time than credible compilation in the tested passes, with thousands of proof lines added for the verified versions.

Coordination is becoming a test target, not just a system design choice. AgentCollabBench injects constraints, tracer strings, false facts, and private strings into 900 software, DevOps, and data-engineering tasks. It finds that communication topology explains 7% to 40% of the variance in multi-hop information survival, with converging DAG nodes losing minority-branch constraints more often than linear chains.

Evolutionary Ensemble of Agents studies coordination during coding-agent search. It keeps scored populations of solver code and agent guidance, then updates agents through Elo-style wins from their solver outputs. On the ICON positional-encoding task, the method finds a rescale-then-interpolate encoding that improves generalization to unseen example counts; at k=10, error stays below 0.15 with 2,000 training steps and below 0.08 with 10,000 steps in the reported setup.