Source note
World-Ego Modeling for Long-Horizon Evolution in Hybrid Embodied Tasks
Embodied World ModelRobot World ModelLong Horizon RolloutNavigation ManipulationVideo Diffusion
Summary
WEM separates long-horizon embodied video prediction into scene-level world state and robot/object ego state. It targets hybrid navigation-manipulation rollouts, where a single video generator tends to lose scene consistency or instruction-conditioned contact dynamics.
Problem
- Existing embodied world models mix persistent scene structure, viewpoint change, robot motion, and contact dynamics in one prediction path, which hurts long-horizon rollouts.
- Hybrid tasks need both navigation scene consistency and manipulation physics across multiple instructions; existing benchmarks focus on short manipulation or single-prompt generation.
- This matters because world models are used for planning, policy simulation, and synthetic robot data.
Approach
- WEM predicts two latent states for each step: a world state from visual history and past instructions, and an ego state from the current instruction plus recent context.
- It defines the world-ego split in three ways, then uses the semantic split by default: robot and manipulated objects are ego; background and inactive objects are world.
- A frozen Qwen3-VL-2B-Instruct state predictor uses 256 queries, split into 192 world queries and 64 ego queries, with role-conditioned attention.
- A Wan2.2-TI2V-5B diffusion transformer is split into a shared preceding expert plus world and ego experts. A predicted semantic mask routes video tokens to the matching expert and recombines them before decoding.
- Training uses flow matching for video latents plus BCE and Dice losses for the world-ego mask.
Results
- The paper introduces HTEWorld: 125K training video clips, more than 4.5M frames, fine-grained action annotations, 300 evaluation trajectories, and more than 2K instructions.
- HTEWorld uses the 16-metric EWMScore from WorldArena and adds 6 metrics for multi-turn and hybrid navigation-manipulation evaluation.
- In the boundary ablation, the semantic world-ego split beats the motion-based split by 2.12 EWMScore points and the intention-based split by 2.79 points.
- WEM is reported to outperform fine-tuned Cosmos-Predict 2.5 2B/14B and WoW-7B on HTEWorld, but the excerpt does not give absolute scores or full baseline margins.
- The paper says WEM remains competitive on existing manipulation-only benchmarks, but the excerpt does not provide the benchmark scores.