Source note
Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos
Vision Language ActionRobot Foundation ModelSynthetic Robot VideosGeometry SupervisionRobot Data Scaling
Summary
GRA adapts VLA models with synthetic robot videos by using those videos for 2D geometry supervision, while training actions only on real robot demonstrations. On three Franka pick-and-place tasks, it raises matched-budget success from 61.1% for Real-only to 68.9% and beats pseudo-action baselines.
Problem
- VLA policies need paired video-action data, but real teleoperation is slow and hardware-specific.
- Human-to-robot video generators can create robot-looking videos, yet they do not provide reliable motor commands.
- Prior pseudo-action methods train the action head on actions recovered from generated pixels; the paper argues this adds control noise and lowers real-robot success.
Approach
- GRA uses generated videos only for spatial geometry supervision.
- It extracts future 2D end-effector waypoints from the source human video using Grounding DINO, SAM2, HaMeR, retargeting, MuJoCo simulation, and calibrated camera projection.
- Stage 1 trains the OpenVLA-OFT vision backbone and a 3-layer MLP 2D waypoint head for 5K steps on 75 generated videos per task; the language model and action head are excluded.
- Stage 2 trains the action head for 10K steps on 25 real demos per task with L1 delta-action loss, while a waypoint loss on real frames keeps the vision features tied to end-effector geometry.
- The action output is a 24-step chunk of 7D delta actions; the waypoint horizon is K=8.
Results
- On 3 real Franka tasks with 30 trials each, GRA reaches 68.9% mean success with 25 real demos and 75 generated videos per task, compared with 61.1% for Real-only, 48.9% for DreamGen-style pseudo-actions, and 54.4% for MimicDreamer-style retargeted pseudo-actions.
- Per task, GRA scores 66.7% on cube→pad, 56.7% on cup→coaster, and 83.3% on mango→plate; Real-only scores 60.0%, 46.7%, and 76.7%.
- The 100-demo Real-only reference reaches 75.6% mean success. GRA cuts the gap to this reference from 14.5 points to 6.7 points while using 4x fewer real demos.
- Frozen-probe diagnostics show generated-to-real transfer is much worse for delta actions than waypoints: waypoint error is 0.41σ, while delta-action error is 1.38σ on the Stage-1 backbone.
- Teacher-forced action evaluation reports GRA at 9.26 mm position error and 0.060 total L1 error, better than Real-only at 11.09 mm and 0.090 and MimicDreamer-style at 13.36 mm and 0.071.
- On cup→coaster ablations, full GRA reaches 56.7% success; removing Stage 1 gives 43.3%, removing the waypoint anchor gives 46.7%, and replacing geometry with delta-action targets gives 36.7%.