Gated handoff from video planning to low-level manipulation control
A practical robot stack now looks more modular at the control boundary: use a video model to draft the approach path, then hand off to a reactive VLA policy for contact and recovery. Veo-Act gives the clearest case for this split. Its Veo-3 planner plus low-level VLA policy lifted average success from 45% to 80% across the tested sim and real manipulation settings, with especially large gains on ambiguous scenes and pass-by interaction. In real pass-by interaction, success moved from 2/13 to 11/13. The paper also gives the limit in plain terms: video prediction can sketch the motion, but action recovery on its own is still too loose for reliable contact-rich execution.
The concrete build here is a gated handoff layer between a trajectory proposer and the action policy you already trust near objects. Teams working on dexterous pick-and-place, cluttered tabletop work, or mobile manipulation with awkward viewpoints can test this without retraining a full end-to-end planner. Start with one class of failures where the current policy chooses the wrong approach under distractors or partial visibility. Log when the robot should stay on the planned path and when it should switch to the reactive controller, then measure whether the handoff improves task completion without adding unsafe contact. The main adoption blocker is not model quality alone; it is the switching logic and the interface between the planner's coarse future and the controller's action chunk format.