Executable harnesses now determine agent cost, reliability, and test-time gains

The harness is becoming an optimization target. TTHE branches and edits executable control programs using runtime errors, tool outputs, public tests, and other unlabeled traces. With DeepSeek-V4-Flash, it raised BIRD accuracy from 12.0% to 50.0% and SWE-bench Verified from 20.0% to 35.0%. The gains are transductive, and imperfect proxy signals can still cause selection errors.

Automated harness adaptation also changes deployment economics. Across 21 task-model pairs, optimized harnesses improved 16 and closed the small language model (SLM) gap on seven. The best SLM agent reached 89.7% of frontier-LLM performance at 4% of the cost. A separate enterprise study encoded source, routing, output, and trace requirements in code; all 270 model-boundary runs passed those contracts, while a bolt-on guardrail reduced measured utility.

Long-Horizon-Terminal-Bench measures sustained execution with 46 containerized tasks and dense subtask rewards. Runs averaged 9.9 million tokens, 231 episodes, and 85.3 minutes. The strongest tested model solved seven tasks at the 0.95 reward threshold. Across all models, the mean pass rate was 4.3%, and timeouts caused 79% of unresolved runs.

Dense grading exposes useful distinctions within failure. Of 690 runs, 433 earned partial reward, including 180 at or above 0.5. Seventy-three runs were near completion, compared with 30 full passes. This gives developers evidence about progress, stalling, and verification failures that final-state scoring hides.

Repository-level coding requires context that matches the procedure being implemented. ProjAgent retrieves functions with similar computational steps even when vocabulary and application domains differ, then uses compiler and static-analysis feedback for repair. It reports 41.14% Pass@1 on REPOCOD, though the available evidence does not quantify its margin over retrieval baselines.

Generation also needs repository-wide invariants after code is written. The Patchwork Problem models imports, calls, dependencies, configuration, schemas, resources, control flow, and routing as coordinated graphs. Its study covers 336 generations and validates the failure categories on 43 real-world AI-generated repositories. The paper does not report precision, recall, or a numerical failure rate in the available excerpt, so its strongest supported contribution is the verifier’s scope and taxonomy.