Recoleta
GitHub
Source note

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

Authors
Zijian Zhang, Yuqing Jiang, Qian Cheng, Si Liu, Ding Zhao, Ping Luo, et al.
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionRobot World Model3d Gaussian SplattingManipulation PolicyFuture Prediction
Open arXivMarkdown source

Summary

GaussianDream adds training-time 3D Gaussian reconstruction and short-horizon future prediction to a VLA robot policy, then uses only the learned prefix tokens at inference.

Problem

  • Standard VLA policies learn mostly from action imitation, so 3D geometry, depth, contact-relevant structure, and short-term scene changes get weak supervision.
  • This matters for manipulation because small geometric errors can move a grasp point, miss a target pose, or fail a spatial relation task.
  • Prior 3D or world-model methods often add depth or point clouds for the current scene, or require heavy prediction rollouts during control.

Approach

  • GaussianDream encodes three temporal context frames, listed as t-10, t-5, and t, into 1024 GaussianDream prefix tokens in the PaliGemma/Gemma-2B prefix space.
  • During training, a static Gaussian head decodes the prefix into a current 3D Gaussian scene, with depth used to back-project 256 × 256 Gaussian centers.
  • A dynamic head predicts future Gaussian center displacements for horizons t+1 through t+5, conditioned on a learned horizon embedding.
  • Training uses RGB rendering loss, depth loss, and pseudo 3D scene-flow loss, plus the base flow-matching action loss.
  • At inference, the Gaussian decoding, rendering, depth, and velocity heads are removed; the policy receives only the learned prefix tokens and samples actions with the base policy.

Results

  • On LIBERO, GaussianDream reports 98.4% average success, compared with π0.5 at 96.7%, GeoPredict at 96.5%, GeoVLA at 97.7%, VLA-4D at 97.4%, 3D-CAVLA at 98.1%, and Spatial Forcing (PyTorch) at 97.6%.
  • On LIBERO sub-suites, it reports 99.0% on Spatial, 99.6% on Object, 99.0% on Goal, and 96.0% on Long; LingBot-VA has a slightly higher LIBERO average at 98.5%.
  • On RoboCasa Human-50, the paper reports 52.6% average success over 24 long-horizon kitchen tasks with 50 trials per task across five scenes.
  • In real-robot evaluation, the paper reports 50.0% success on tasks covering attribute grounding, spatial relations, stacking and unstacking, and long-horizon goals.
  • The reported efficiency claim is that inference needs no Gaussian rendering, no future video rollout, and no separate planner, because only prefix tokens condition the action policy.

Link