Recoleta
GitHub
Source note

Being-H0.7: A Latent World-Action Model from Egocentric Videos

Authors
Hao Luo, Wanpeng Zhang, Yicheng Feng, Sipeng Zheng, Haiweng Xu, Chaoyi Xu, et al.
Source
arXiv
Published
Collection
Embodied AI
Latent World ModelVision Language ActionRobot Foundation ModelEgocentric VideoRobot Data Scaling
Open arXivMarkdown source

Summary

Being-H0.7 trains a robot policy to use future video information during training while acting from current observations at test time. It targets the gap between direct VLA policies and video-rollout world-action models by moving future reasoning into latent tokens.

Problem

  • Direct VLA policies can learn shortcut mappings from observations to actions because action labels are sparse compared with visual input.
  • Video-based world-action models add future prediction, but pixel rollout costs training and inference compute and may model visual detail that does not affect control.
  • The problem matters for long-horizon and dynamic manipulation where contact, object motion, and task progress affect the next action.

Approach

  • The model inserts K=16 learnable latent queries between the multimodal context and the action chunk, so the Transformer builds a compact action-oriented state before predicting actions.
  • Training uses two matched branches: a prior branch sees current instruction, H=4 observation frames, state, and latent queries; a posterior branch replaces those queries with embeddings from future observations.
  • Future frames are encoded with a frozen ViT and Perceiver resampler into K embeddings, then aligned with the prior latent states over the last L=9 Transformer layers.
  • Both branches train with a flow-matching action objective over T=20 action chunks, plus latent alignment and norm/rank regularizers to reduce latent collapse.
  • At inference, the posterior branch is removed, so the policy predicts actions without generating future images.

Results

  • The excerpt claims state-of-the-art or comparable performance across 6 simulation benchmarks, but the provided text does not include the table values, metrics, or success rates.
  • Real-world evaluation is reported on 3 robot platforms and 12 tasks covering dynamic scenes, physical reasoning, motion reasoning, long-horizon execution, and generalization.
  • The paper claims Being-H0.7 leads all 5 ability-oriented real-world suites, with examples including catching a fast rolling ball, pouring into a moving container, folding garments, conveyor package sorting, and hammering a nail.
  • Deployment with latency-aware universal asynchronous chunking is reported at 3-4 ms/step for Being-H variants, with no test-time future-frame generation.

Link