Source note
Open-H-Embodiment: A Large-Scale Dataset for Enabling Foundation Models in Medical Robotics
Summary
Open-H-Embodiment is a large open medical robotics dataset built to support foundation models across many robot platforms and tasks. The paper uses it to train a surgical vision-language-action policy and an action-conditioned world model, then shows gains on suturing and cross-platform evaluation.
Problem
- Medical robot learning lacks large open datasets with synchronized video and robot kinematics, which blocks foundation-model training in surgery and related healthcare tasks.
- Earlier datasets are small and narrow: JIGSAWS has about 3 hours, Expanded CRCD about 7 hours, SutureBot about 6 hours, and ImitateCholec about 20 hours, each on a single platform.
- This matters because surgical policies need to handle deformable tissue, long task sequences, and strict error tolerance, and general robot VLAs trained on non-medical data have performed poorly on surgical benchmarks.
Approach
- The authors build Open-H-Embodiment, a cross-embodiment dataset with 770 hours, 124,019 episodes, 119 datasets, 49+ institutions, 20 robot platforms, and 33 task families. Each dataset pairs video with synchronized kinematics.
- The dataset covers surgery, robotic ultrasound, and endoscopy across simulation, benchtop, ex vivo, in vivo, and clinical settings. It includes major systems such as da Vinci, dVRK, Versius, MIRA, BiTrack, and Maestro.
- They post-train NVIDIA's GR00T-N1.6 on Open-H to create GR00T-H, described as the first open foundation vision-language-action model for medical robotics.
- They also fine-tune Cosmos-Predict 2.5 into Cosmos-H-Surgical-Simulator, a 2B-parameter action-conditioned world model that takes one frame plus kinematic actions and predicts future surgical video across nine robotic platforms from one checkpoint.
Results
- Dataset scale: Open-H has 770 hours versus prior public surgical kinematic datasets at about 3h, 7h, 6h, and 20h. It spans 20 platforms; prior listed datasets each span 1 platform.
- End-to-end suturing (SutureBot): GR00T-H completed 5/20 trials (25%). ACT, GR00T-N1.6, and LingBot-VA each completed 0/20. At the throw stage, GR00T-H retained 12/20 trials versus 4/20 for ACT, 4/20 for GR00T-N1.6, and 1/20 for LingBot-VA.
- Out-of-distribution generalization (unseen wound + lighting shift): GR00T-H reached 54% average subtask success, compared with 30% for GR00T-N1.6 and 5% for ACT. On pickup and handover, it scored 70% vs. 40% for GR00T-N1.6; on throw and extract, 42.5% vs. 5%.
- Data efficiency on SutureBot: with 33% of fine-tuning data (about 2h), GR00T-H and ACT were both about 47% average success, while GR00T-N1.6 was about 20%. With 100% data (about 6h), GR00T-H reached about 73%, ACT about 50%, and GR00T-N1.6 about 37%.
- Cross-embodiment evaluation: GR00T-H beat the base GR00T-N1.6 across dVRK-Si, Versius, and MIRA, with the paper reporting a statistically significant gain in overall average success (p < 0.001). Training data for these platform tests included 5.2 hours on Versius, 22 minutes on MIRA, and 6 hours on dVRK-Si.
- Ex vivo suturing on pork belly: GR00T-H achieved 64% average success across 29 subtasks (n=290 total, 10 trials per subtask). Several subtasks hit 10/10, including needle pickup, set-down needle, and all three knot-tying steps, while cutting subtasks were weaker at 2/10 and 3/10.
- World model: the paper claims the first multi-embodiment surgical action-conditioned world model, trained on data from 9 platforms and 32 datasets. It reports evaluation with L1 and SSIM over 72 generated frames and 3 seeds, but the excerpt does not provide the actual numeric metric values.