Source note
Multi-View Video Diffusion Policy: A 3D Spatio-Temporal-Aware Video Action Model
Summary
MV-VDP is a robot manipulation policy that predicts both future multi-view videos and action heatmaps, so the model tracks 3D scene structure and how the scene will change after an action. The paper targets low-data manipulation and reports strong gains over video-action, 3D, and vision-language-action baselines in simulation and on real robots.
Problem
- Many robot policies learn from 2D images while acting in 3D space, which creates a mismatch between observation and control and raises the amount of robot data needed.
- Many policies reuse image-text backbones trained on static images, so they do a poor job modeling scene dynamics and action consequences over time.
- This matters most in low-data manipulation: the paper focuses on settings with only 5 to 10 demonstrations per task, where common behavior cloning and VLA systems often fail.
Approach
- The method projects cropped point clouds into three fixed orthographic views, producing multi-view RGB images and multi-view heatmaps for the robot end-effector pose.
- It uses the same representation type for pretraining-compatible video inputs and for action prediction: the model predicts future RGB videos and future heatmap videos together.
- The backbone is a 5B Wan2.2 video diffusion model, extended with cross-view attention so tokens from different camera views interact at each timestep.
- Predicted heatmap peaks from the views are back-projected into 3D end-effector positions. A separate 170M decoder uses denoised video latents to predict rotation and gripper state, then combines position, rotation, and gripper outputs into action chunks.
- Training uses diffusion loss on both video and heatmap sequences, plus LoRA fine-tuning and SE(3) augmentation. The paper says full fine-tuning did not improve performance.
Results
- Meta-World, 7 tasks, 5 demos per task, 25 trials per task: MV-VDP reaches 89.1% average success, ahead of Track2Act 67.4%, DreamZero 61.1%, AVDC 58.9%, DP 37.7%, BC-R3M 35.4%, BC-Scratch 26.2%, and UniPi 11.4%.
- Meta-World per-task examples: MV-VDP gets 25/25 on D-Open, D-Close, Btn, and Handle, 24/25 on Btn-Top and Fct-Open, and 8/25 on Fct-Cls.
- Real world, about 10 demos per task, 10 trials per task: on basic tasks MV-VDP reports 10/10 on Put Lion, 4/10 on Push-T, and 7/10 on Scoop Tortilla. Baselines shown in the excerpt are weaker: BridgeVLA 9/10, 0/10, 4/10; UVA 2/10, 0/10, 0/10; DP3 0/10, 0/10, 0/10; π0.5 1/10, 0/10, 0/10.
- Real-world unseen variants: the excerpt reports 5/10 on Put-B and 6/10 on Put-H for MV-VDP, showing some transfer to changed background and object height. The table is truncated, so the remaining unseen-task numbers and final average are not fully visible in the provided text.
- The paper also claims strong robustness to hyperparameters, including keeping good performance with only 1 diffusion step, but the excerpt does not provide the supporting table or exact numbers.
- It claims realistic future-video prediction and interpretable action previews, but the excerpt does not include a quantitative video-prediction metric.