Source note
TrajAudit: Automated Failure Diagnosis for Agentic Coding Systems
Summary
TrajAudit diagnoses failed repository-level coding-agent runs by finding the earliest step that sent the agent down the wrong path. It combines test-error hints, trajectory filtering, and tool-based inspection to handle long, noisy execution traces.
Problem
- Coding agents fail on multi-file software maintenance tasks, and developers need to know the first decisive error step so they can fix the agent, prompt, tools, or workflow.
- Existing LLM diagnosis methods read whole trajectories or fixed chunks, so accuracy drops on repository-level traces with 20 to more than 100 steps and observations that can exceed 70% of the content.
- Prior benchmarks understate this difficulty; Who&When averages 22.24 steps and 1,384.11 characters per step, while RootSE averages 50.94 steps and 5,830.71 characters per step.
Approach
- TrajAudit first uses failed test code and error messages to produce a preliminary diagnosis, including likely failure phases and reasons.
- It compresses trajectory observations through semantic saliency folding: content without failure-related patterns or keywords is replaced with a folded marker, while likely useful code, patches, and error signals stay visible.
- An investigator agent reads the task, the preliminary diagnosis, and the folded trajectory, then calls APIs to inspect hidden observation content only when needed.
- The output is the predicted earliest decisive error step plus a natural-language justification.
- The paper also introduces RootSE, a benchmark with real failed coding-agent trajectories from SWE-agent, OpenHands, and AutoCodeRover on SWE-bench and SWE-bench Pro tasks.
Results
- On RootSE, TrajAudit beats the strongest existing baseline by more than 24.4 percentage points in localization accuracy.
- TrajAudit uses at least 18% fewer tokens than baselines.
- RootSE contains 93 failed execution instances, more than 4,500 execution steps, and about 27 million characters.
- RootSE task descriptions average 8,223.51 characters, compared with 240.47 for Who&When; RootSE tasks modify 2.9 files on average, compared with 1.7.
- RootSE covers 3 programming languages; Who&When covers 1.
- Annotation quality reports Cohen’s kappa of 0.78 for earliest decisive error step identification, followed by 100% final consensus after arbitration.