Recoleta
GitHub
Source note

Towards Predictive, Aligned, and Scalable Robot Learning

Authors
Peijun Tang, Shangjin Xie, Baifu Huang, Binyan Sun, Haotian Yang, Kuncheng Luo, et al.
Source
arXiv
Published
Collection
Embodied AI
Robot Foundation ModelWorld ModelVision Language ActionRobot Data ScalingDexterous Manipulation
Open arXivMarkdown source

Summary

Lumo-2 is a latent world-action model that predicts action-relevant future dynamics before generating robot actions. Its three-stage modality alignment process targets better temporal reasoning, physical understanding, scaling, and dexterous manipulation.

Problem

  • Robot policies often map a single observation directly to actions, which creates ambiguity when the same view occurs at different phases of a task such as pouring.
  • Reconstruction-focused action tokenization can produce accurate action codes that remain poorly aligned with visual context, language, and downstream control.
  • This matters because weak temporal and cross-modal alignment limits generalization to long-horizon, physically complex, and cross-embodiment tasks.

Approach

  • Lumo-2 first infers a compact latent representation of physically grounded world dynamics from visual observations, language, proprioception, and a short history buffer, then conditions action generation on that representation.
  • Stage 1 jointly trains visual world-dynamics and action autoencoding with a vector-quantized architecture, using visual dynamics to guide action reconstruction and actions to regularize the dynamics representation.
  • Stage 2 adds semantic action tokens and aligns them with vision and language through action reconstruction, behavior description, vision-language-guided action generation, cross-modal prediction, and contrastive objectives.
  • Stage 3 co-trains vision-language understanding, future projection, and action generation so the model projects latent future dynamics before producing action chunks.
  • Training uses Qwen3.5-4B, diverse robot data across multiple platforms, egocentric videos, and a temporal context buffer to reduce partial-observability errors.

Results

  • The excerpt provides no numerical task success rates, benchmark scores, dataset totals, or baseline margins.
  • Lumo-2 claims consistent gains over strong VLA and WAM baselines across 3 real-world task categories: temporal reasoning, physical understanding, and high-control-complexity manipulation.
  • The strongest claimed improvements occur on long-horizon and dexterous manipulation tasks that require future-state prediction and physical reasoning.
  • The training recipe includes 3 progressive alignment stages and uses 30,000 steps on 64 NVIDIA H100 GPUs, but these are training details rather than performance results.
  • The model can be fine-tuned with egocentric human videos and VisionPro data, according to the excerpt.

Link