---
source: arxiv
url: http://arxiv.org/abs/2604.19092v1
published_at: '2026-04-21T05:09:56'
authors:
- Feng Jiang
- Yang Chen
- Kyle Xu
- Yuchen Liu
- Haifeng Wang
- Zhenhao Shen
- Jasper Lu
- Shengze Huang
- Yuanfei Wang
- Chen Xie
- Ruihai Wu
topics:
- world-model-benchmark
- robotic-manipulation
- embodied-evaluation
- video-world-models
- sim2real
relevance_score: 0.94
run_id: materialize-outputs
language_code: en
---

# RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

## Summary
RoboWM-Bench is a benchmark for testing whether robot manipulation videos generated by world models can be turned into executable actions that actually complete tasks. The paper shows that strong visual video generation does not reliably produce physically executable manipulation behavior.

## Problem
- Existing video world model benchmarks focus on visual realism, semantics, or diagnostic physical plausibility, but they do not directly test whether generated manipulation behavior can be executed by a robot.
- This matters because robot learning from imagined or generated videos only works if the predicted behavior matches real dynamics, contact, and task constraints.
- Real-robot evaluation is costly and hard to reproduce, so the paper targets a scalable and repeatable embodiment-grounded benchmark.

## Approach
- The benchmark takes an initial scene observation and task description, asks a world model to generate a future manipulation video, then converts that video into robot actions and executes them in simulation.
- For human-hand videos, it estimates 3D hand poses with HaMeR and retargets them to robot end-effector motions, with a modified pose/orientation formulation and gripper signal for more stable contact geometry.
- For robot videos, it uses an inverse dynamics model trained first in simulation and then fine-tuned on a small amount of real Franka data to recover joint actions from consecutive frames.
- It reconstructs real scenes in a high-fidelity real-to-sim setup built on LeHome, then checks both step-level nodes such as contact or lift and final task success.
- The task suite covers rigid, articulated, deformable, long-horizon, and bimanual manipulation scenarios.

## Results
- On human task-level evaluation, Wan 2.6 is the strongest reported model: 83% on Pick Object, 100% on Push Button, 80% on Pour Water, 80% on Open Drawer, 80% on Put in Drawer, and 40% on Fold Towel. Weaker models often collapse on harder tasks, such as Cosmos with 0% on Pour Water and Fold Towel.
- Human step-level results show that early contact is much easier than full execution. For Put on Plate, several models reach 100% contact, but final place success drops to 15% for Cosmos, 55% for Wan 2.2, 70% for Wan 2.6, 30% for Veo 3.1, and 70% for LVP.
- On robot task-level evaluation, success is much lower. Without task-specific fine-tuning, the best reported scores are Wan 2.6 with 50% on Close Drawer, 40% on Push Object, 40% on Push Button, 20% on Pick Object and Put on Plate, and 0% on Pull Object and Put in Drawer.
- Fine-tuning helps but does not solve the problem. Cosmos-FT reaches 90% on Close Drawer, 60% on Push Button, 50% on Pick Object and Push Object, 40% on Put on Plate and Pull Object, 30% on Discard Trash, and 20% on Put in Drawer.
- Robot step-level results expose where failures happen. In Put in Drawer, Cosmos-FT gets 60% contact but only 20% on lift, above drawer, in drawer, and close drawer, while non-fine-tuned baselines are often 0% after the first step.
- The paper’s main empirical claim is that visual realism and executability diverge: current video world models still fail on spatial reasoning, contact stability, long-horizon execution, and deformable-object dynamics, even after manipulation fine-tuning.

## Link
- [http://arxiv.org/abs/2604.19092v1](http://arxiv.org/abs/2604.19092v1)