Recoleta
GitHub
Source note

Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

Authors
Junhao Shi, Siyin Wang, Xiaopeng Yu, Li Ji, Jingjing Gong, Xipeng Qiu
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionRobot Data ScalingGeneralist Robot PolicyInverse DynamicsTask Agnostic Pretraining
Open arXivMarkdown source

Summary

TAP pretrains a VLA on task-agnostic robot motion before language-conditioned behavior cloning. The main claim is that cheap unlabeled interaction data can teach low-level physical control and reduce dependence on expert teleoperation.

Problem

  • VLA training usually needs observation-language-action triples collected by human teleoperation, which is expensive and slow to scale.
  • Many robot trajectories without useful task labels are discarded, even though they contain contact, grasping, pushing, and object-motion information.
  • This matters because generalist robot policies need more physical interaction data than teams can collect with annotated expert demonstrations alone.

Approach

  • TAP uses task-agnostic trajectories from two sources: unrelated Bridge trajectories in simulation and autonomous random play on a WidowX 250s robot.
  • Stage 1 trains with inverse dynamics: given observation o_t and next observation o_{t+1}, the model predicts the 7D delta-pose action a_t that caused the change.
  • The inverse-dynamics target makes the model attend to moving hands and objects instead of static background pixels.
  • Stage 2 finetunes the same backbone and action head with a small set of language-labeled expert demonstrations, using standard behavior cloning.
  • The real-world random-play pipeline builds a safe pose library, samples reachable waypoints, adds a contact heuristic, injects bounded Gaussian noise, and records the resulting trajectories.

Results

  • In SIMPLER, TAP-20k reaches 33.32% Avg-All success, compared with 23.15% for the same architecture trained with standard behavior cloning, 7.75% for OpenVLA, and 3.03% for RT-1-X.
  • SIMPLER Avg-Partial success for TAP-20k is 45.82%, compared with 31.79% for standard behavior cloning, 42.30% for Octo, and 53.10% for π0.
  • SIMPLER Avg-Entire success for TAP-20k is 20.82%, compared with 14.50% for standard behavior cloning, 20.33% for Octo, and 27.05% for π0.
  • More task-agnostic pretraining improves SIMPLER Avg-All success across 8k, 14k, and 20k episodes: 24.47%, 30.21%, and 33.32%.
  • In real-world WidowX tests with 30 hours of random play and 200 expert demonstrations per task, TAP averages 28% on carrot-on-plate versus 9% for training from scratch and 36% for NORA.
  • On the real-world push-pumpkin task, TAP averages 61% versus 21% for training from scratch and 56% for NORA; under background texture shift it scores 65% versus 0% and 55%, and under viewpoint variation it scores 25% versus 0% and 0%.

Link