Agent reliability is an engineering control problem

Several items treat a large language model (LLM) agent as one part of a controlled runtime. STORM rejects writes when an agent has read stale files, then returns fresh context so the agent can retry. The production-agent architecture paper makes the same point at the tool boundary: a model proposes, deterministic code verifies, accepted actions are committed, and rejected actions get typed feedback. Capframe gives a concrete Model Context Protocol (MCP) version of this idea with scoped capability tokens and deterministic policy checks for tool calls.

The shared concern is practical. Parallel agents can overwrite each other or act on old assumptions. Tool-using agents can receive too much authority. The strongest work in this group adds version checks, commit rules, denial receipts, and audit records around the model.

OpenComputer and AgentAtlas both argue for evaluation below the final success number. OpenComputer builds 1,000 desktop tasks across 33 applications and scores them with programmatic checks tied to real application state, including browser profiles, files, databases, and saved documents. That matters because screenshot scoring can miss hidden state errors.

AgentAtlas looks at the trajectory itself. It labels control decisions such as Act, Ask, Refuse, Stop, Confirm, and Recover, then shows that prompt format and evaluation axis can change model rankings. Its audit also finds that memory, state, and efficiency receive weak coverage in existing agent benchmarks. The evidence says agent evaluation needs task results plus trace-level checks.

optimize_anything broadens LLM-based search into a common loop: edit a text artifact, score it, pass diagnostic side information back to the proposer, and try another candidate. The same API is applied to prompts, code, agent structures, scheduling policies, CUDA kernels, and numerical solvers. The reported gains are strongest when evaluators return useful side information such as traces, profiler data, costs, or images.

DIFFCODEGEN applies a narrower form of runtime evidence to code selection. It samples multiple candidate programs, fuzzes inputs, compares observed behavior, clusters candidates, and returns the medoid of the largest behavior group. This avoids extra LLM calls after generation and reports much lower time and token use than prior test-time selection methods that need public tests or model-based judges.

Two reliability papers focus on when code models should act. The defer-and-recover paper calibrates correctness scores, accepts outputs above a threshold, and sends uncertain cases to validation or recovery steps such as compiler checks, static analysis, prompt augmentation, and task decomposition. Its results show better Brier score and expected calibration error on MBPP+ and defect prediction, but also warn that no single uncertainty metric worked across tasks.

The input-adaptation paper tests a different control point. It rewrites or adjusts inputs at inference time when a validity score is low. Early results are clearest for vulnerability detection: CodeBERT improves from 63.36% accuracy to 76.75% with latent transformation, while existing uncertainty metrics remain weak error detectors. The lesson is narrow and useful: code automation needs task-specific confidence handling, not a generic confidence number.