Recoleta
GitHub
Source note

MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models

Authors
Pablo Valle, Shaukat Ali, Aitor Arrieta, Lionel Briand
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionRobot EvaluationTest OraclesFailure LocalizationMulti Agent Generation
Open arXivMarkdown source

Summary

MANGO automatically generates fine-grained test oracles for VLA robot tasks from natural-language instructions. It improves evaluation and debugging by checking ordered atomic steps such as open, pick, place, and close, rather than only the final simulator state.

Problem

  • VLA robot benchmarks often use hand-written symbolic oracles that check only the final state, such as whether an object is inside a container at the end of execution.
  • These oracles require domain effort, depend on benchmark-specific formats such as Python or .bddl files, and give weak debugging signal for long-horizon manipulation tasks.
  • A robot may fail by missing an intermediate step, dropping an object, or doing steps in the wrong order; a final-state oracle usually reports only task failure.

Approach

  • MANGO builds a reusable atomic task library from complex natural-language tasks, with parameterized actions such as Open(fridge), Pick(bottle), Place(bottle, fridge), and Close(fridge).
  • It maps each atomic task to simulator checks using available simulator functions, such as open state, holding state, contact, and spatial relations.
  • For each full instruction, it decomposes the instruction into an ordered or partially ordered sequence of atomic tasks and attaches the matching oracle to each step.
  • Three agent roles run iterative checks: the Generator creates candidate task libraries and oracles, Assessor agents check logic, object grounding, function use, and execution reliability, and the Judge accepts the candidate or sends correction instructions.
  • Each generation loop can stop when accepted or after 10 iterations.

Results

  • The paper evaluates MANGO on 2 benchmarks: LIBERO_10 and RoboCasa Humanoid Tabletop.
  • The authors claim MANGO generates executable fine-grained oracles that detect a similar number of failures as hand-written symbolic oracles.
  • The generated oracles also localize failures by identifying the failed atomic step and order violations in the task sequence.
  • The paper reports ablation studies on component contributions and on reducing the initial task set while keeping oracle quality, but this excerpt gives no quantitative ablation values.
  • The provided excerpt does not include exact success rates, failure counts, precision, recall, runtime, or baseline numbers beyond the 2 named benchmarks and the 10-iteration cap.

Link