Source note
How VLAs (Really) Work In Open-World Environments
Summary
This paper argues that current BEHAVIOR1K evaluation can overstate how well vision-language-action models work in open-world household tasks because it scores only final task state and ignores unsafe execution. The authors audit top B1K systems, show large instability and frequent safety violations, and propose safety-aware metrics that penalize harmful behavior during the task.
Problem
- BEHAVIOR1K mainly uses success rate and Q-score, which check final sub-goal satisfaction but ignore how the robot reached that state.
- That misses safety-critical failures such as dropping objects, colliding with furniture, damaging support objects, or finishing sub-goals in unsafe ways.
- For long-horizon household tasks, final-state scoring also gives weak diagnostic signal because failures can come from many stages: task understanding, navigation, grasping, manipulation, or action ordering.
Approach
- The paper analyzes the top two B1K 2025 Challenge policies, RLC and Comet, on the 50 challenge tasks with 10 randomized trials each.
- It studies robustness in two ways: reproducibility of official results using released checkpoints, and consistency across trials for the same task.
- It uses expert video review on 500 task recordings by 8 robotics experts to label failure causes in 10 categories, including grasp failure, collision, task confusion, navigation failure, and execution order confusion.
- It proposes two safety-aware metrics on top of B1K Q-score: sQ, which penalizes unsafe handling or bad placement of target objects, and seQ, which also adds sub-goals for support non-target objects and penalizes hazards such as dropping them or moving them more than 10 cm.
- The safety rules are concrete: for example, target objects can be penalized if placed with roll or pitch more than 30 degrees from upright, and support objects are marked violated if they fall or are displaced too far.
Results
- Reproducing the official RLC checkpoint showed large gaps from posted results on many tasks, with differences above 27% on tasks 24, 26, and 27; for task 37, the reproduced score was 0.1 while the posted score was 0.0.
- The qualitative audit covered 500 recordings (50 tasks × 10 trials) reviewed by 8 experts. The paper reports that grasp failure was the most common error category, with collisions also frequent, plus cases of task confusion such as putting chicken on a shoe rack or a shoe in a fridge.
- On the 20-task table shown, average RLC performance drops from Q = 0.256 to sQ = 0.239, and average Comet drops from Q = 0.192 to sQ = 0.173, showing that safety penalties reduce apparent performance even before accounting for support-object damage.
- With the broader safety-enhanced metric, average scores are seQ = 0.356 for RLC and 0.304 for Comet, compared with seQ-Oracle = 0.395 and 0.338. The gap reflects violations on support objects and target handling that plain Q-score does not capture.
- The same table reports average violation counts of nTV = 0.35 and TV = 0.53 for RLC, versus nTV = 0.31 and TV = 0.59 for Comet, indicating both non-target and target safety issues are common.
- Several task-level examples show large safety impact: for RLC on Bring water, Q = 0.233 drops to sQ = 0.133 with TV = 1.2; on Rearrange kitchen, Q = 0.275 drops to sQ = 0.175; for Comet on Bring water, Q = 0.400 drops to sQ = 0.267; on Set up coffee, Q = 0.200 drops to sQ = 0.150 with TV = 1.1.