Agent work is being judged by complete loops, visible failures, and auditable evidence

Large language model (LLM) agent evaluation is getting closer to full engineering work. The Meta-Agent Challenge gives a coding agent a sandbox, APIs, a development split, and a hidden verifier, then asks it to build another agent. The best visible results roughly match human baselines on some domains, yet performance varies by model and task, with clear gaps on science QA.

TeleSWEBench adds a domain-specific stress test. It builds 734 tasks from real srsRAN 5G commits and requires file localization before functional scoring. The strongest automated software engineering tools reach up to 25% ship-ready changes, and localization drops sharply as prompts give less edit detail. This makes the bottleneck concrete: repository-scale, protocol-heavy code still defeats many current agents.

Several papers treat agent success as a problem of usable evidence. The process taxonomy scores six high-traction AI development support tools across specification, context, roles, execution, validation, and portability. Its clearest finding is uneven coverage: richer artifacts improve traceability, while portability across agents becomes harder.

Context-as-a-Service gives a more concrete mechanism. It lets an agent query indexed source files, tests, examples, and documentation during documentation review. In two production-SDK case studies, it raised retained findings from 5 to 13 and cut wall-clock time and input tokens. Self-reflective APIs make a related point at the API boundary: structured repair suggestions can help agents recover from validation failures more reliably than longer prose errors, at least for the Anthropic models tested.

Agent reliability depends on facts that the model cannot infer at planning time. The Model Context Protocol (MCP) study measures description-code inconsistency across 19,200 tool pairs from 2,214 servers. DCIChecker reports inconsistency in 9.93% of pairs, including omitted behavior, overclaimed capability, and hidden side effects. These errors can mislead tool choice and create security blind spots.

Ekka addresses a different hidden failure mode in LLM serving. It compares a target serving engine against a reference implementation at intermediate model states, then ranks the component where outputs first diverge. On real vLLM and SGLang silent errors, it reports 80% pass@1 and 88% pass@5 diagnosis accuracy. The result matters because output-only checks often miss the failing layer: model code, kernel backend, numerical precision, or serving logic.

The operational papers focus on what an agent can remember and query during repeated work. UModel turns observability data into linked objects for services, pods, hosts, metrics, logs, traces, events, runbooks, and tools. On the AIOps 2025 Challenge dataset, the paper reports an 8% gain in root cause localization over a naive agent approach, and it describes more than one year of Alibaba Cloud deployment.

The stigmergy proposal applies memory to tool choice. It stores tools, MCP tools, and skills as nodes in a local graph, with decayed success and failure evidence on transitions. The evidence is still early: the article gives platform token-cost motivation and an implemented design, while the central controlled token-reduction test remains open. Its useful contribution is the explicit idea that agents should carry outcome history into future capability selection.