Source note
ToolMisuseBench: An Offline Deterministic Benchmark for Tool Misuse and Recovery in Agentic Systems
Summary
ToolMisuseBench is an offline benchmark for testing how tool-using agents fail and recover when tool calls break in specific ways. Its main contribution is deterministic, replayable fault injection with explicit limits on steps, calls, and retries, so reliability can be measured in a controlled way.
Problem
- Tool-using agents often fail for operational reasons such as invalid arguments, schema drift, timeouts, rate limits, weak retry logic, and authorization errors, even when their language understanding is strong.
- Existing agent benchmarks usually measure broad capability, but they do not isolate tool misuse and recovery under fixed budgets with exact replay of the same failures.
- This matters for deployment because agents can look capable in static tests and still fail repeatedly once strict tool schemas, policy rules, and retry limits are enforced.
Approach
- The paper introduces a deterministic benchmark where each task includes an instruction, tool schemas, initial state, success criteria, a fault plan, and budget limits for steps, tool calls, and retries.
- It uses offline simulators across four domains: CRUD, retrieval, file, and scheduling. Faults are injected in replayable ways, including schema drift, rate limit, timeout, authorization failure, and adversarial error rewriting.
- Evaluation tracks more than final task completion. It measures success, invalid call behavior, policy violations, recovery success, tool-call efficiency, budget overruns, and budgeted success under call caps.
- The released package includes data generation, evaluation, trace logging, aggregate reports, and support for both built-in baselines and external agents through a simple reset/act interface.
- The dataset has 6,800 tasks with fixed splits: 5,000 train, 800 development, and 1,000 public test.
Results
- On the 1,000-task public test split, all three baselines reach the same overall task success: 0.250. The heuristic baseline averages 2.95 tool calls, while schema-repair and policy-aware baselines average 3.25 calls.
- Overall policy violations are 0.168 for the heuristic baseline and 0.166 for both schema-repair and policy-aware baselines. Overall recovery is 0.000 for heuristic and 0.250 for the other two baselines.
- For timeout faults, success is 0.499 for heuristic and 0.502 for schema-repair and policy-aware. Recovery on timeout rises from 0.000 with heuristic to 0.502 with the repair-based methods.
- For schema drift faults, success is 0.503 for heuristic and 0.497 for schema-repair and policy-aware. Recovery on schema drift rises from 0.000 with heuristic to 0.497 with the repair-based methods.
- For authorization and rate limit subsets, all baselines have 0.000 success in the released setting.
- Budgeted success curves are flat with AUC 0.25 for all agents, which the paper interprets as evidence that better recovery heuristics alone do not fix the main bottleneck under the current fault mix and budget constraints.