Source note
CRAFT: Video Diffusion for Bimanual Robot Data Generation
Summary
CRAFT generates photorealistic, action-labeled bimanual robot demonstrations by running trajectories in a simulator and converting them into videos with a Canny-guided video diffusion model. It targets data scarcity and weak visual diversity in bimanual imitation learning, especially for sim2real transfer, viewpoint changes, and cross-embodiment training.
Problem
- Bimanual imitation learning needs many demonstrations, but real teleoperation data is expensive and usually covers limited viewpoints, object layouts, lighting, and robot embodiments.
- Existing augmentation methods usually change only one factor, such as viewpoint or embodiment, and often do not produce new action labels paired with the new visuals.
- This matters because dual-arm tasks depend on precise gripper-object contact and coordination, so narrow training data hurts robustness and transfer.
Approach
- CRAFT starts from a small real dataset, builds a digital twin, replays and expands trajectories in simulation, and keeps successful new trajectories as extra supervision.
- It renders simulator videos, extracts Canny-edge control videos, and feeds those edges into a pre-trained video diffusion model together with a real reference image and a language instruction.
- The key idea is that Canny edges keep the motion and object structure that matter for manipulation while dropping simulator texture details, which gives the diffusion model room to generate realistic appearance.
- The generated videos inherit action labels from the simulator trajectories, so the method outputs action-consistent demonstrations rather than image-only edits.
- The same pipeline supports seven augmentation types: object pose, lighting, object color, background, cross-embodiment transfer, camera viewpoint, and joint wrist plus third-person multi-view generation.
Results
- In simulation cross-embodiment transfer from bimanual UR5 to bimanual Franka, CRAFT (Ours) reaches 82.6% on Lift Pot, 89.3% on Place Cans, and 86.0% on Stack Bowls, using 1000 generated demos and no target-robot demos.
- On the same simulation tasks, CRAFT (Target) with no extra generated data gets 11.3%, 6.0%, and 21.6%, while Shadow gets 2.0%, 2.3%, and 6.0%.
- The paper reports that CRAFT (Ours) beats the target-robot collected-data baseline, which scores 55.0%, 69.0%, and 59.0% on those three simulation tasks, while requiring 100 collected target demos.
- In real-world cross-embodiment transfer from bimanual xArm7 to bimanual Franka, CRAFT (Ours) achieves 17/20, 15/20, and 16/20 successes on LR, PC, and SB, compared with 4/20, 1/20, and 2/20 for CRAFT (Target), 2/20, 1/20, and 1/20 for Shadow, and 5/20, 2/20, and 3/20 for collected target data.
- In an ablation on Stack Bowls, Canny guidance improves success from 10.3% without Canny to 21.6% with Canny; the collected-demo upper bound is 59.0%.
- The excerpt does not give full quantitative results for all seven augmentation types, but it claims consistent gains over existing augmentation baselines and over straightforward data scaling across simulated and real bimanual tasks.