Source note
From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws
Summary
HarnessFix repairs LLM agent harnesses by tracing failed executions to specific runtime steps and harness layers, then generating scoped patches. The paper claims 15.2%–50.0% held-out performance gains over initial harnesses across SWE-Bench Verified, Terminal-Bench 2.0 Verified, GAIA, and AppWorld.
Problem
- LLM agents fail for reasons outside the base model, including tool schemas, context assembly, orchestration, logging, validation, sandboxing, and policy checks.
- Existing self-improvement methods often optimize prompts or workflows from final scores, so they can change the wrong part of the agent harness or make broad edits.
- Reliable repair matters because long-horizon software, terminal, research, and app-automation agents depend on external tools and state changes, where small harness flaws can cause bad final submissions.
Approach
- HarnessFix converts raw traces and harness code into HTIR, a step-level representation with request/response messages, roles, execution status, artifact or state effects, provenance links, and control-flow links.
- A diagnosis agent uses HTIR to locate the responsible TraceStep or steps, map them to ETCLOVG layers, and write diagnosis records.
- Similar diagnosis records are grouped into recurring flaw records before any edit is made.
- A repair agent maps each flaw record to scoped repair operators, such as argument validation, context refresh, verification-gated finalization, or state-delta logging.
- A validation agent checks that a patch stays within scope, reduces the target flaw, and avoids unacceptable regressions.
Results
- On 4 benchmarks, SWE-Bench Verified, Terminal-Bench 2.0 Verified, GAIA, and AppWorld, HarnessFix improves held-out test performance over the initial harnesses by 15.2%–50.0%.
- The paper says HarnessFix beats human-designed harnesses and self-evolution baselines, but the excerpt does not provide exact baseline scores.
- The method uses 4 cooperating LLM agents: trace abstraction, diagnosis, repair, and validation.
- The analysis covers 7 harness layers in the ETCLOVG taxonomy: Execution, Tooling, Context, Lifecycle, Observability, Verification, and Governance.
- Ablations test prompt-only repair, trace-grounded diagnosis, scoped repair operators, and regression-aware acceptance; the excerpt says every component helps, without giving ablation numbers.