Recoleta
GitHub
Source note

Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents

Authors
Chenyu Zhao, Shenglin Zhang, Yihang Lin, Wenwei Gu, Zhimin Chen, Yongqian Sun, et al.
Source
arXiv
Published
Collection
Software Intelligence
Software Engineering AgentsAgent RecoveryCode IntelligenceTelemetryFailure Diagnosis
Open arXivMarkdown source

Summary

PROBE improves post-failure recovery for software engineering agents by turning failed-run telemetry into evidence-grounded retry guidance. It targets failed agent runs in code repair, enterprise workflows, and AIOps service mitigation.

Problem

  • Software engineering agents fail after tool errors, unchecked outcomes, wrong workflow state, or premature submission; manual recovery is slow and inconsistent.
  • A final benchmark failure label does not show which step broke, which evidence matters, or what the next attempt should change.
  • In 257 unresolved first attempts, 172 cases (66.93%) came from insufficient validation, tool/subprocess failure handling, or state/workflow error.

Approach

  • PROBE records span-level telemetry from failed runs, including metrics, logs, traces, agent intent, tool-environment state, and optional evaluator outcomes.
  • It localizes failure evidence with metric anomaly checks using MAD-based z-scores, empirical quantiles, Isolation Forest, log signatures, repeated tool failures, trace patterns, and evaluator signals.
  • It fuses related findings into structured evidence records with anchors, time scope, severity, support, conflicts, and provenance.
  • An anchor-first diagnosis step emits structured fields such as failure anchor, primary cause, behavioral mistake, contributing factors, evidence, and confidence.
  • A Guidance Gate produces retry guidance with a target, operation, verification signal, and boundary condition only when the diagnosis is evidence-grounded, actionable, and within agent-side control.

Results

  • The evaluation covers 257 initially unresolved cases across SWE-bench, EnterpriseOps-Gym, and AIOpsLab: 102, 106, and 49 cases respectively.
  • PROBE reports 65.37% Top-1 diagnosis accuracy, 43.58 percentage points above the strongest non-PROBE baseline.
  • PROBE reports a 21.79% recovery rate, 12.45 percentage points above the strongest non-PROBE baseline.
  • The paper reports a diagnosis-recovery gap: 65.37% diagnosis accuracy versus 21.79% recovery, so correct diagnosis often does not translate into a successful next run.
  • A Microsoft IcM prototype attaches PROBE as a side channel with 0 required changes to the agent policy, toolset, or execution budget; the excerpt gives no quantitative IcM success metric.

Link