Executable coding evidence

Teams building customer-support, ops, or browser agents can now justify an evaluation harness that records the whole run and grades state changes, tool use, and user-visible artifacts. The case is stronger than transcript review alone. The sim/eval paper lays out a concrete stack: separate scenario design, simulation, and grading; log world state, tool traces, transcripts, and screenshots or telemetry; prefer deterministic checks for end state and procedure; use model judges only for narrow questions. CUJBench adds a hard operational example. It freezes browser and backend evidence into deterministic incident snapshots, and six frontier models still reach only 19.7% accuracy, with cross-modal synthesis as the main failure. A practical build here is a replayable episode runner for one narrow workflow such as refunds, order changes, or failed checkout diagnosis, with pass/fail gates on backend state and required tool calls. A cheap first check is to take ten recent support or incident cases, replay them in a sandbox, and compare transcript-only scores against outcome and tool-trace checks. If the rankings differ, the current eval stack is missing failures that matter in production.

Repository coding agents need an environment-setup stage with its own success metric before any patch benchmark or deployment workflow. RAT reports that automated setup can succeed on a large share of real repositories, with ESSR of 63.2% on Python, 41.3% on Java, 98.7% on Rust, and 68.7% on JS/TS across a 2,000-plus repository benchmark. The open-source commit study shows why this stage should stay separate from patch generation. Even on small real commits in C projects, generated changes still fail to compile, trigger static-analysis warnings, and miss tests. That points to a concrete product gap: a repository intake worker that detects language, selects a base image, installs dependencies, runs smoke tests, and emits a machine-readable readiness report before any code-edit agent starts. The first users are internal developer-tools teams and benchmark builders who waste time on tasks that fail before execution begins. A cheap validation step is to run the intake worker across a mixed set of fifty repositories and measure how often it reaches a reproducible build or test command without manual fixes.

Scientific coding teams can test a code-generation workflow built for cases where no input/output test set exists yet. MOSAIC gives a concrete pattern: use domain-specific teacher examples to produce rationale templates and pseudocode, keep a compact rolling context of prior function signatures and summaries, and limit automatic debugging to syntax and import repair. On SciCode, that setup improves results across GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash, and the compact context window matters because keeping all prior code drops performance sharply in the ablation. This is useful for labs and research software groups that write numerical or simulation code from method descriptions, where validation often depends on domain review more than fixed test answers. A practical pilot is a scaffold that asks a domain lead to curate a small teacher set for one subfield, then generates workflow code with explicit intermediate steps and handoff points for human review. The first check is simple: compare generated code quality on a small internal task set with and without the compact context and teacher examples, then see whether reviewers spend less time correcting algorithm structure.