Execution-checked agent workflows

Repository repair agents can start treating tests as editable evidence, not fixed gatekeepers. Agent-CoEvo is the clearest case for a build that runs code patches and test patches together, scores both in one execution matrix, and keeps only candidates that explain the issue report and fail on the buggy repo before they pass on the repaired one. The operational pain is familiar: teams hit bug reports where the existing tests are incomplete, stale, or too weak to pin down the behavior change, so a code-only repair loop either overfits to bad tests or stalls.

A practical product shape is a CI-side repair worker for maintainers of large Python or Java repositories. It would open a draft PR with two linked diffs: the implementation patch and the test changes that justify it. The cheap check is narrow and clear: sample issues where maintainers had to edit tests during the fix, then compare a code-only agent against a coevolution loop on merged rate and reviewer rejection rate. The paper reports 41.33% resolved on SWE-bench Lite and 46.4% on SWT-bench Lite, with higher test quality than the listed baselines, which is enough to justify testing this workflow on repos with steady bug backlog and existing CI coverage.

High-volume document and transaction workflows can move model use to build time and run validated code in production. Compiled AI gives a concrete pattern: generate a small business-logic function once inside a fixed template, run security scanning, type and syntax checks, sandbox tests, and golden-set accuracy checks, then execute the approved artifact with no runtime model call for the main path. The user pressure is simple: operations teams in billing, prior authorization, claims intake, and document processing need repeatable outputs, short latency, and an audit trail that survives review.

This is buildable now as an internal tool for workflow engineers who already maintain prompt-based automations. Start with one narrow process that has a stable schema and enough traffic to matter, such as invoice field extraction or rules-based intake routing. The first validation is cost and variance: compile the current prompt workflow into code, replay a week of production inputs, and compare per-transaction latency, output drift, and manual exception rate. The reported break-even point is about 17 transactions on BFCL, with 4.5 ms median latency, 100% reproducibility, and large cost differences at scale. The DocILE result also suggests a bounded hybrid mode is workable when messy documents still require a constrained model call.

Teams testing microservices can replace brittle recorded mocks with an online dependency simulator that answers requests during the test run and keeps state across the scenario. Mirage shows why this matters: held-out scenarios break static substitutes because the artifact has to guess all relevant behavior ahead of time, while the runtime simulator can react to new parameters, error paths, and multi-step flows. This maps to a common blocker in service teams where downstream systems are expensive to provision, unavailable in CI, or too unstable for repeatable integration tests.

The concrete build is a test harness that stands up a FastAPI-compatible mock endpoint backed by an LLM, seeded with dependency source code when available and traces when it is not. Early users are platform and backend teams with service meshes, many internal APIs, and frequent contract drift. The first check is direct: pick one dependency that currently relies on record-replay fixtures, run the same integration suite against the real dependency, the current mock, and an online simulator, then compare pass-fail agreement and payload-shape fidelity. Mirage reports 99% status-code fidelity and 99% response-shape fidelity in white-box mode across 110 scenarios, while record-replay reached 62% and 16% on the same benchmarks. That gap is large enough to justify a focused pilot around the most failure-prone dependencies.