AI engineering research is tightening evidence gates around agents and model dependencies

C2VEval shows how a vision-to-code benchmark can overstate capability when task text leaks the answer. In circuit-to-Verilog generation, model headers such as sum, cout, or fsm_3state let multimodal LLMs (MLLMs) produce plausible register-transfer-level code without reading the diagram. When the authors replace the image with a blank one, Mirage mode matches or beats real-image mode on all eight evaluated MLLMs. After anonymizing identifiers, GPT-5.4 drops from 45.51% to 24.55% Functional Pass@1, and Opus 4.6 drops from 52.69% to 11.38%.

Claw-Eval-Live applies the same evidence-first instinct to workflow agents. It grades 105 tasks using traces, audit logs, service state, post-run files, command traces, and tests. Claude Opus 4.6 leads at 66.7% pass rate, and no evaluated model reaches 70%. The result is a concrete ceiling on today’s workflow automation claims, especially for HR, management, and multi-system business tasks.

IronCurtain frames vulnerability discovery as a stateful investigation. A central orchestrator reads an append-only journal, dispatches specialist agents, and keeps code search, harness creation, and validation outside a single model context. The workflow reproduced the 1998 OpenBSD TCP SACK vulnerability and used Opus 4.6 to build a fuzzer that narrowed the trigger to a two-sequence-number difference at a 32-bit signed integer boundary. A QEMU-based driver then reproduced the kernel panic.

The same report gives useful cost and access details. Opus and Sonnet runs used about 10 million tokens per investigation, with reported costs near $150 and $30. Hosted GLM 5.1 runs averaged 27 million tokens and still produced a proof-of-concept plus sanitizer-validated harness for an 18-year-old integer truncation flaw. The evidence favors security pipelines that validate reachability, not static bug reports alone.

The LLM supply-chain paper argues for deployer-side compatibility gates before hosted model updates reach production. Its examples are practical: unit-test contracts, valid JSON requirements, code-only output rules, and risk categories such as authentication, data validation, and structured output. In a small validation over seven Claude models and 25 prompts, structured JSON tasks showed more drift than SQL and authentication tasks. One backend SQL function passed with Sonnet 4, then failed a safe-encoding test the next day.

Deptex extends the same operational lens to open-source package risk. It combines an organization graph, policy checks, code-property-graph reachability, and constrained LLM verification. In its scenario, a CVSS 9.8 issue appears in 10 repositories, but eight paths are downgraded because they sit in offline batch scripts six function hops deep, while two public unauthenticated API paths receive high exposure. The paper is mostly a design and scenario report, so its claims need measured alert-reduction results.

The hardware items share a concern about models that produce convincing artifacts without enough domain grounding. VeriGround addresses that problem in circuit generation with anonymized training, blank-image refusal examples, and decision-focused preference tuning. It reaches 46.11% Functional Pass@1 on normal C2VEval inputs and 42.51% on anonymized inputs, while keeping false refusals low on valid examples.

JuliaHub’s Dyad 3.0 announcement makes a related product claim for physical-system engineering. It connects agents with physics simulation, control design, safety analysis, and embedded code generation. The release says a Scientific Machine Learning digital twin used four sensor inputs to predict pump faults with over 90% accuracy in work with Binnies and Williams Grand Prix Technologies. The announcement includes useful engineering direction, but its benchmark evidence is limited compared with the circuit-generation paper.