Recoleta
GitHub
Source note

A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

Authors
Shuo Wang, Hanyuan Xu, Yingdong Hu, Fanqi Lin, Yang Gao
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionSim2realRobot EvaluationRobot Foundation ModelsPolicy Ranking
Open arXivMarkdown source

Summary

This paper studies when simulation gives the same VLA policy evaluation decisions as real robot tests. It finds that REALM tracks real-world rankings best among three simulators, and that limited simulator fine-tuning improves sim-real alignment.

Problem

  • Real-world VLA evaluation is expensive: the study reports 1,115 physical rollouts versus 11,800 simulated rollouts.
  • Existing simulators can look realistic while still giving wrong policy rankings or wrong failure patterns.
  • Policy developers need simulation to support model selection and diagnosis under vision, layout, language, and behavior changes.

Approach

  • The authors align 9 tabletop manipulation tasks between simulation and a DROID real-robot setup.
  • They evaluate 5 VLA policies: π0, π0-FAST, π0.5, GR00T N1.6, and GR00T N1.7.
  • They compare VLA-Arena, SIMPLER, and REALM using Spearman rank correlation, Pearson correlation, and Mean Maximum Rank Violation against real-world results.
  • They measure perturbation sensitivity by normalizing each policy’s success-rate drop across vision, layout, language, and behavior perturbations.
  • They test REALM-based fine-tuning with different amounts of simulator data to see how adaptation changes sim-real alignment.

Results

  • REALM gives the strongest average policy-ranking correlation: Spearman 0.700, Pearson 0.785, MMRV 0.030; VLA-Arena gets 0.575/0.725/0.060, and SIMPLER gets 0.400/0.402/0.128.
  • REALM matches the real-world perturbation severity order across all 4 dimensions: behavior is highest sensitivity at 1.000, layout is mid-level at 0.644 versus 0.679 real, and vision is lowest at 0.000 versus 0.008 real.
  • Simulator post-training in REALM raises proxy Spearman correlation from 0.700 to 0.875 and Pearson correlation from 0.785 to 0.878.
  • Post-training cuts proxy MMRV from 0.030 to 0.015 and sensitivity MAE from 0.110 to 0.041.
  • The data-scaling result is non-monotonic: Tune-5 improves several metrics, Tune-10 gives the best overall alignment, and Tune-20 reduces perturbation-sensitivity alignment below the untuned REALM setting.
  • Object replacement in REALM changes absolute success rates on the banana task but keeps the same policy ordering across 5 replacement objects: corn, zucchini, hotdog, carrot, and cucumber.

Link