Source note
From World Models to World Action Models: A Concise Tutorial for Robotics
Summary
This tutorial argues that robot world models should be defined as action-conditioned predictors of future observations or states. It adds the idea of world action models, where predicted futures are linked to executable robot actions.
Problem
- Robotics papers use the term "world model" for many different systems, including latent dynamics models, video predictors, physics simulators, and action-conditioned generative models, which makes comparison hard.
- A predictor that shows a likely future does not by itself tell a robot which action to execute, so visual prediction needs a path to control.
- The choice of prediction target, observation type, and action type affects visual fidelity, spatial structure, physical meaning, and control use.
Approach
- The paper defines a world as the task-relevant robot, objects, and environment, then defines a world model as a model that predicts future observations or states conditioned on observation history and action.
- It splits world models into 2 main classes: observation-space models that predict future observations, and state-space models that predict compact or structured states.
- It organizes observation-space models by 2 axes: observation spatial explicitness, such as RGB, RGB-D, and point clouds, and action abstraction, such as low-level robot commands, latent actions, and language instructions.
- It organizes state-space models by the state type: latent vectors, point tracks, neural-symbolic predicates, and physical variables.
- It defines world action models as policies that model future observations and action sequences together, then groups them into 4 paradigms: imagine-then-execute, video-feature-conditioned action prediction, joint video-action modeling, and auxiliary video prediction for policy learning.
Results
- The excerpt provides no new quantitative benchmark results, accuracy numbers, success rates, or dataset-scale measurements.
- The main concrete output is a 2-part taxonomy of world models: observation-space prediction and state-space prediction.
- The observation-space taxonomy uses 4 observation levels: RGB, multi-view RGB, RGB-D, and point clouds.
- The action-conditioning taxonomy uses 4 action levels: low-level robot actions, interface actions, latent actions, and language instructions.
- The state-space taxonomy lists 4 state choices: latent states, point tracks, neural-symbolic predicates, and physical states.
- The world action model taxonomy lists 4 ways to connect predicted futures to robot actions: 2-stage visual subgoal execution, feature-conditioned action prediction, joint video-action generation, and auxiliary video prediction during policy training.