Source note
TTHE: Test-Time Harness Evolution
Summary
TTHE adapts an LLM agent's executable harness during evaluation by rewriting and selecting harness programs from unlabeled execution traces. Across code, SQL, software engineering, and data-science tasks, it improves a fixed ReAct-style harness without changing model weights or using gold labels during adaptation.
Problem
- LLM agents often use a fixed control program for context construction, tool use, verification, and failure recovery, so they cannot adjust when test-time failures differ from development failures.
- Existing prompt, workflow, and model adaptation methods usually run before evaluation, update model parameters, revise one response, or require labeled feedback.
- The problem matters because execution traces contain useful signals such as runtime errors, test results, tool outputs, and malformed intermediate artifacts, but these signals do not directly reveal task correctness.
Approach
- TTHE treats the harness, rather than the model weights, as the state that adapts. The harness is executable Python code around a frozen LLM.
- For each unlabeled test batch, it creates multiple harness branches, runs them, and records prompts, completions, tool calls, outputs, errors, artifacts, and runtime states.
- Agentic proposers edit their own branch using these traces and proxy signals such as execution health, round-trip consistency, and public-test pass rate.
- An agentic judge selects one final branch without seeing gold answers, hidden tests, or reference outputs. The selected harness persists to the next batch.
- The method uses the same frozen backbone for solving, proposing, and judging, so adaptation occurs through code changes rather than weight updates or a separately trained adaptation model.
Results
- On DeepSeek-V4-Flash, TTHE improved BIRD from 12.0% to 50.0%, LiveCodeBench from 30.0% to 38.3%, SWE-bench Verified from 20.0% to 35.0%, and DS-1000 from 38.0% to 44.0% against the fixed ReAct-style baseline.
- On BIRD, the method improved MiMo V2.5 from 32.0% to 52.0% and Kimi K2.5 from 28.0% to 48.0%.
- The experiments cover five execution-grounded domains, including text-to-SQL, competitive programming, software engineering, data-science coding, and agentic tool use; the excerpt gives exact headline numbers for four domains.
- The gains are transductive: each batch is measured with the harness selected using that batch's unlabeled traces, while gold labels are used only for post-selection evaluation.
- Ablations and trace audits identify non-monotonic search-budget effects, limited candidate coverage, selection regret, and judge errors caused by imperfect execution proxies as key limitations.