Source note
RoboFlow4D: A Lightweight Flow World Model Toward Real-Time Flow-Guided Robotic Manipulation
Robot World ModelFlow Guided ManipulationVision Language Action3d Motion PlanningReal Time Robot Control
Summary
RoboFlow4D predicts multi-frame 3D gripper flows from RGB images and text, then uses those flows to guide a robot policy in a closed loop. It targets real-time manipulation by replacing multi-model 3D flow pipelines with one lightweight diffusion transformer model.
Problem
- 2D flow planners operate in image space, so they miss depth and geometry needed for collision-free manipulation.
- Recent 3D flow planners often chain video generation, depth, grounding, and point tracking models, which adds latency and memory cost.
- Real robots need low-latency planning that can update from new observations during a task.
Approach
- RoboFlow4D takes recent RGB frames, an optional set of 2D query points, and a language instruction as input.
- DINOv2 and SigLIP encode vision and text; a 3D Perceiver learns 3D-aware features by aligning with VGGT features.
- A diffusion transformer called FlowDiT denoises future 3D trajectories and outputs multi-frame flows with shape
keypoints × frames × 3D position. - The action policy receives both normal state inputs and an encoded flow plan, so it can track the predicted motion.
- Control runs as a slow-fast loop: RoboFlow4D plans lower-frequency atomic-task trajectories, while the policy executes higher-frequency action chunks.
Results
- On LIBERO with Diffusion Policy, average success rises from 78.9% to 85.1%, a +6.2 point gain; suite gains are +8.2 Spatial, +1.7 Object, +6.8 Goal, and +8.0 Long.
- On LIBERO with a DiT policy, average success rises from 83.7% to 87.7%, a +4.0 point gain; suite gains are +6.0 Spatial, +0.7 Object, +3.0 Goal, and +6.4 Long.
- The paper reports a +11.0 point average gain on ManiSkill3 and shows DP improving from 12.3% to 22.0% average on the visible PushCube, PickCube, and StackCube table rows.
- Real-world tasks improve by 5 to 20 percentage points according to the paper excerpt.
- The authors claim under 1 second planning latency, a 120× speedup over modular flow pipelines, and more than 24% smaller model scale than other flow models.