Source note
Veo-Act: How Far Can Frontier Video Models Advance Generalizable Robot Manipulation?
Summary
Veo-Act tests whether a frontier video generator, Veo-3, can improve robot manipulation without task demonstrations. The paper finds that video prediction alone gives good high-level plans but weak low-level control, and a hierarchical video-planner-plus-VLA system raises success rates on dexterous pick-and-place tasks.
Problem
- The paper studies generalizable robot manipulation in open-ended scenes, where current vision-language-action policies still fail on object ambiguity, viewpoint changes, and contact-heavy dexterous control.
- Pure video-to-action pipelines can keep the generalization of large video models, but their action recovery is too imprecise for reliable contact-rich manipulation.
- This matters because collecting expert robot data is expensive, and a system that can use web-scale video priors with little or no demonstration data could improve robot generalization at lower data cost.
Approach
- The authors first test a zero-shot pipeline: Veo-3 generates a future video from the current image and language instruction, and an inverse dynamics model (IDM) converts frame-to-frame changes into robot actions.
- The IDM is trained from random-play data rather than expert demonstrations. It uses a DINOv3 visual encoder and two heads: one predicts actions, the other predicts a gate value for whether the robot is in an interaction phase.
- Veo-Act uses the generated video as a high-level motion plan, turns it into an action chunk, smooths the chunk, and executes it until the gate detector says the robot should switch to a reactive low-level VLA policy.
- During execution, the system can switch back from the VLA policy to the remaining planned action sequence after the interaction phase, pruning the part of the plan that overlaps with the gated interaction window.
- Training data for the IDM includes 300k simulation frame pairs, 100k extra random-motion simulation samples, and 150k real-world samples, with STEM-OB augmentation to reduce the sim-to-real gap.
Results
- Main headline: the paper reports that Veo-Act improves the average success rate of the strong VLA baseline (\pi_{0.5}) from 45% to 80% across the simulated and real dexterous-hand environments they test.
- In simulation, under the wrist-camera invisible experimental setting, overall task success improves from 10/30 = 0.33 for (\pi_{0.5}) to 20/30 = 0.67 for Veo-Act. Instruction-following improves from 11/30 = 0.37 to 25/30 = 0.83.
- In simulation, under similar-object distractors, overall success improves from 12/30 = 0.40 to 28/30 = 0.93. Under pass-by interaction, overall success improves from 0/30 = 0.00 to 14/30 = 0.47.
- In real robot tests, under similar-object distractors, overall success improves from 8/16 = 0.50 to 12/16 = 0.75. Under pass-by interaction, it improves from 2/13 = 0.15 to 11/13 = 0.85.
- In real robot tests with richer semantics, overall success improves from 2/19 = 0.11 to 15/19 = 0.79, and instruction-following improves from 4/19 = 0.21 to 18/19 = 0.95.
- The zero-shot Veo-3+IDM baseline shows partial capability but weaker control: in simulation, the video-based baseline VPP reaches 15/30 = 0.50 overall in wrist-camera-invisible experimental settings, 6/30 = 0.20 with similar-object distractors, and 1/30 = 0.03 in pass-by interaction. This supports the paper's claim that current video models produce roughly correct task trajectories but lack precise low-level control.