Source note
LIBERO-Safety: A Comprehensive Benchmark for Physical and Semantic Safety in Vision-Language-Action Models
Summary
LIBERO-Safety is a benchmark and dataset for testing whether VLA robot policies can finish manipulation tasks without unsafe contact or unsafe instruction following. It adds parametric safety scenarios, generated collision-free demonstrations, and a cross-model evaluation that shows current VLAs still fail often under safety constraints.
Problem
- Current VLA benchmarks test task completion more than safe execution, so a policy can look strong while colliding with clutter, human hands, or held objects.
- This matters because deployment near people requires collision control, safe hand-object interaction, and refusal of harmful commands.
- Human teleoperation limits data scale; the paper reports 7.4 minutes per task for teleoperation, which slows collection of safety data.
Approach
- The benchmark defines safety tasks with UBDDL, an extension of BDDL that samples scene layouts, object poses, camera settings, visual variation, sensor noise, dynamic entities, and safety predicates.
- It splits safety into 5 suites: affordance-aware grasping, human-robot interaction, tabletop spatial avoidance, free-space hand-object avoidance, and semantic safety reasoning. Each suite has 3 levels, L0-L2.
- A human gives sparse object-centric keyposes. CuRobo turns those keyposes into full robot motions while checking collisions at every timestep.
- The data generator creates multiple spatial variants per keypose and samples combinations, so one human keypose script can produce many demonstrations.
Results
- The benchmark contains 75 tasks: 5 suites × 3 levels × 5 tasks. The generated scenes include 7,603 unique scenes, 953 objects, and 462 hand-object pairs.
- The dataset has 19,664 human-screened, collision-free demonstrations across 40 physical-safety training tasks. The authors omit all L2 tasks and the semantic reasoning suite from training to keep those tests unseen.
- Data collection effort drops from 7.4 to 1.8 minutes per task versus human teleoperation in Table 2, with planner collision checks replacing human-dependent safety checks.
- The evaluation covers 8 VLA policies for physical safety and 2 embodied foundation models for semantic refusal, using 10 trials per task and 3 random seeds for physical rollouts.
- OpenVLA-OFT reaches 79.3% SR on AAG-L1, 80.0% on HRI-L1, 41.3% on TSA-L1, and 50.7% on FSHOA-L1, but falls to 1.3% on AAG-L2 and 42.7% on FSHOA-L2.
- pi0.5 is the strongest named baseline in the shown table: 84.7% SR on HRI-L0, 88.7% on HRI-L1, 83.3% on HRI-L2, and 51.3% on FSHOA-L2. It still has large gaps on hard physical safety, including 35.3% on AAG-L2.