Agent Workflow Optimization and Validation

Teams running repetitive workflows such as budget approval should test a small language model with a task-specific harness before committing to frontier-model inference costs. The harness optimizer can inspect failed trajectories and edit instructions, tool availability, context selection, hooks, and orchestration loops. In the reported budget-approval task, an adapted Gemma-4-26B-A4B setup reached 98.3% accuracy, up from 75.0% with the default harness and above the 97.3% reported for Gemini-3.1-Pro. Across 21 task-model pairs, adaptation improved 16 and closed the measured gap on seven.

A practical adoption gate is a replay set drawn from one high-volume workflow, split by instance diversity and operational edge cases. Compare the frontier agent, the small model with its current harness, and the adapted small-model agent on task accuracy, contract violations, latency, and cost per completed case. Keep a frontier-model fallback for cases outside the validated workflow. The study found larger gains on repetitive tasks and stronger small models, so a diverse holdout is needed before using the savings estimate for capacity planning.

Agent operators can run candidate harness edits in shadow mode using traces already produced by SQL, coding, and data workflows. Each branch should record prompts, tool calls, outputs, runtime errors, artifacts, and recovery decisions. A proposer edits the executable harness; a selector scores candidates with execution health, round-trip consistency, or public-test results. Promotion should require a separate labeled canary set because these proxies can select a worse branch.

TTHE shows the size of the possible gain with a frozen model: on DeepSeek-V4-Flash, BIRD rose from 12.0% to 50.0% and SWE-bench Verified from 20.0% to 35.0%. The same experiments found selection regret, judge errors, limited candidate coverage, and non-monotonic returns from larger search budgets. An initial deployment can therefore cap branches and compute, retain every code diff, and promote only when both proxy scores and canary outcomes improve. Rollback remains a normal code release because the adapted state is the harness program.

Developers evaluating terminal agents should add executable graders for meaningful intermediate states: environment setup, artifact creation, test completion, schema validity, and final verification. Record reward by checkpoint alongside elapsed time, episode count, token use, retries, and the last verified state. This separates agents that stall early, make substantial progress, or finish the work but fail final validation.

Long-Horizon-Terminal-Bench found partial reward in 433 of 690 runs, including 180 runs at or above 0.5. Seventy-three runs scored between 0.75 and 0.95, compared with 30 full passes at the 0.95 threshold. Timeouts accounted for 79% of unresolved runs, while an average run consumed 9.9 million tokens, 231 episodes, and 85.3 minutes. A cheap first check is to decompose five representative internal tasks into weighted checkpoints and rerun the current agent under the existing time limit. The resulting traces will show whether engineering effort belongs in planning, state recovery, verification, or runtime budgeting.