Trend

Executable control and verification now set the ceiling for coding agents

Week · 2026-W28 · Software Intelligence

Coding Agents Agent Harnesses Software Verification Long Horizon Evaluation Repository Workflows Context Engineering

Overview

This week, coding-agent progress depended on the control layer around the large language model (LLM): executable harnesses, runtime checks, and repository workflows. TTHE posted large gains with frozen model weights, while long-horizon evaluation exposed severe completion limits. Product designs adopted similar controls, but comparative evidence remained sparse.

Findings

Adaptive harnesses and long-horizon evaluation

Test-Time Harness Evolution (TTHE) rewrites and selects executable control programs using unlabeled execution traces. With DeepSeek-V4-Flash, it raised SWE-bench Verified performance from 20.0% to 35.0% and BIRD from 12.0% to 50.0%, while keeping model weights fixed. The method still depends on imperfect proxy signals and can select a weaker harness.

Long-Horizon-Terminal-Bench shows the remaining operational limit. Across 46 containerized tasks, agents averaged 9.9 million tokens and 85.3 minutes per task. The best tested model solved 15.2% at the 0.95 reward threshold, and timeouts caused 79% of unresolved runs. Dense subtask grading made partial progress visible and gave harness designers better failure evidence than a final pass rate alone.

Sources

TTHE: Test-Time Harness Evolution
Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu, Yike Guo, Xinmei Tian, Bo Han
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, LeoweiLiang

Executable specifications and paired verification

Repository tasks are gaining stronger intermediate artifacts. ReProAgent converts issue reports into fail-to-pass reproduction tests through bug localization, root-cause analysis, repository retrieval, and runtime validation. It reproduced 58.43% of SWT-bench-lite issues and 70.30% of SWT-bench-verified issues at an average cost of $0.14 per case.

DualVeri applies reusable property templates to Apache Spark and checks each property with both Lean 4 proofs and property-based tests against PySpark. Templates increased proof synthesis success by 1.6× on average, reduced proof hallucinations by 59%, and cut testing synthesis cost by 3.8× on average. Disagreements between proofs and tests also exposed gaps between the formal model and the running implementation.

Sources

ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reports
Quanjun Zhang, Yi Zheng, Ye Shang, Weifeng Sun, Haichuan Hu, Chunrong Fang, Zhenyu Chen, Liang Xiao
Agentic Proof and Property-Based Testing via Property-Templates in Data-Intensive Computing
Seongmin Lee, Yaoxuan Wu, Miryung Kim

Context, workflow, and cost controls

Operational tooling is making agent inputs and actions inspectable before deployment. ContextOps statically checks context redundancy, density, structure, and source concentration, then exposes command-line and continuous integration (CI) gates. Its published example estimated 12% token savings, but the project reports no accuracy or false-positive comparison.

OneDev places agents inside issues, isolated workspaces, pull requests, review, and CI. Avriz applies shadow evaluation, traffic caps, and model-tier ceilings to a learned router facing a 12× output-token price spread. These designs specify useful controls, yet neither source provides aggregate quality or cost gains against a baseline. The production ideas are ahead of the evidence.