Recoleta
GitHub
Source note

WorldSample: Closed-loop Real-robot RL with World Modelling

Authors
Yuquan Xue, Le Xu, Zeyi Liu, Zhenyu Wu, Zhengyi Gu, Xinyang Song, et al.
Source
arXiv
Published
Collection
Embodied AI
Real Robot RLWorld ModelRobot Data AugmentationSim2realDexterous Manipulation
Open arXivMarkdown source

Summary

WorldSample trains real-robot RL policies with extra transitions generated by an action-conditioned world model, with real rollouts used to keep generation tied to the task. It targets lower physical rollout cost for manipulation tasks that need contact handling and precise alignment.

Problem

  • Real-robot RL can improve beyond demonstrations, but each physical rollout costs time, can wear hardware, may need human reset, and records only one action-outcome path.
  • Imitation learning can fail in states missing from demonstrations, while pure generated rollouts can add visual or contact errors that corrupt critic targets.
  • This matters for contact-rich manipulation because failures and recovery states are costly to collect but useful for learning.

Approach

  • Start from physical robot rollouts, then sample locally perturbed action sequences around the executed actions.
  • Use an action-conditioned Cosmos-Predict2.5 world model, adapted on demonstrations and online rollouts, to generate future video observations.
  • Label generated trajectories with a reward model, then add them to RL training as synthetic transitions.
  • Use Policy-Paced Learning: Q-aware selection balances positive and negative generated samples, and an entropy-based schedule reduces synthetic data when the actor is uncertain.
  • Run video generation and world-model post-training asynchronously so robot control and policy updates keep running.

Results

  • On five real manipulation tasks, average success was 82% for WorldSample, compared with 56% for HIL-SERL, 69% for WMPO, and 64% for VLAW.
  • Average training steps fell from 56K with HIL-SERL to 23K with WorldSample, a 59% reduction. Average training time fell from 83 min to 64 min.
  • Per-task WorldSample success was 95% on Pushing, 95% on Insertion, 95% on Sorting, 84% on Pick & Place, and 42% on Assemble. HIL-SERL reached 84%, 63%, 66%, 55%, and 10% on the same tasks.
  • World-model quality improved over demo-only adaptation: demo-only PSNR/SSIM/LPIPS was 10.27/0.428/0.809, while dual-view demo+rollout adaptation reached 29.89/0.925/0.035.
  • On the insertion ablation, full WorldSample reached 95% success in 10K steps and 30 min. Without scheduling it reached 61% in 18K steps and 48 min; without Q-selection it reached 76% in 12K steps and 36 min; without full PPL it reached 86% in 32K steps and 69 min.

Link