Execution Checks for Coding Agents

Coding assistants that already sample multiple solutions should add a sandboxed execution stage before returning one answer. The concrete build is simple: generate a small set of diverse inputs, run each candidate with timeouts, record outputs, exception types, and timeouts as an execution fingerprint, then choose from the largest all-success behavior cluster.

Semantic Voting gives the clearest case for this as an adoption change. Across 18 HumanEval+ and MBPP+ configurations, execution-based selectors beat output-pattern majority voting by 19 to 52 percentage points, and sketch-based generated inputs were the best input source in its ablation. Sketch-and-Verify adds a useful candidate-generation rule for cheap model tiers: ask for distinct algorithmic sketches, fill each sketch several times, and verify the resulting candidates by execution. On 19 hard HumanEval+ problems for Gemini 3.1 Flash Lite, K=2,M=5 solved 11 problems, compared with 5 for flat N=10 sampling.

A practical first test is to run this selector on recent internal coding-assistant tasks where multiple candidates were already produced. Track pass rate, sandbox cost, and cases where no all-success cluster appears. DSDE can add a risk score when the top candidate’s behavior cluster is far from the alternatives, giving reviewers a reference-free signal before full validation.

Software engineering agents need a retry record that explains the failed run at the span level. A useful implementation would store tool calls, logs, traces, agent intent, tool-environment state, evaluator outcomes, and repeated failure patterns, then produce a bounded retry instruction with a target, operation, verification signal, and boundary condition.

PROBE shows why this belongs beside the agent as an operational side channel. In 257 unresolved first attempts across SWE-bench, EnterpriseOps-Gym, and AIOpsLab, 66.93% of cases came from insufficient validation, tool or subprocess failure handling, or state and workflow error. PROBE reported 65.37% Top-1 diagnosis accuracy and 21.79% recovery, with a Microsoft IcM prototype attached without changing the agent policy, toolset, or execution budget.

The adoption blocker is the vague failed-run artifact: a final benchmark failure or incident label rarely tells the next run what to change. Teams running repo repair agents, service mitigation agents, or enterprise workflow agents can start by logging spans for failed tasks and measuring whether retry prompts tied to a specific failure anchor recover more cases than generic reruns.

Teams maintaining agent skill libraries should turn operational assumptions inside each skill into checked contracts. The build is a skill scanner that extracts package versions, imports, URLs, API paths, environment variables, Docker images, GitHub Actions, CLI flags, and config files; labels each mention as operational or incidental; validates the operational mentions against live sources; and opens a localized repair prompt or pull request when a contract fails.

SkillGuard is a concrete template for this maintenance layer. DriftBench includes controlled drifts, real-world drifts, and negative controls. Contract-free CI probes produced 40% false positives, while SkillGuard reported 0 false positives over 599 no-drift and hard-negative cases. In a live scan of 49 real skills, it reached 86% conservative precision and 55% recall, and one-round contract-guided repair reached 78% success.

This is most useful for long-lived skills that call external services, install packages, configure infrastructure, or depend on authentication flows. The first deployment can run in report-only mode on a skill library, compare alerts against recent skill failures, and check whether localized drift reports reduce repair time.