---
source: arxiv
url: https://arxiv.org/abs/2607.02431v1
published_at: '2026-07-02T17:00:37'
authors:
- Yuquan Xue
- Le Xu
- Zeyi Liu
- Zhenyu Wu
- Zhengyi Gu
- Xinyang Song
- Bofang Jia
- Ziwei Wang
topics:
- real-robot-rl
- world-model
- robot-data-augmentation
- sim2real
- dexterous-manipulation
- policy-learning
relevance_score: 0.9
run_id: materialize-outputs
language_code: en
---

# WorldSample: Closed-loop Real-robot RL with World Modelling

## Summary
WorldSample trains real-robot RL policies with extra transitions generated by an action-conditioned world model, with real rollouts used to keep generation tied to the task. It targets lower physical rollout cost for manipulation tasks that need contact handling and precise alignment.

## Problem
- Real-robot RL can improve beyond demonstrations, but each physical rollout costs time, can wear hardware, may need human reset, and records only one action-outcome path.
- Imitation learning can fail in states missing from demonstrations, while pure generated rollouts can add visual or contact errors that corrupt critic targets.
- This matters for contact-rich manipulation because failures and recovery states are costly to collect but useful for learning.

## Approach
- Start from physical robot rollouts, then sample locally perturbed action sequences around the executed actions.
- Use an action-conditioned Cosmos-Predict2.5 world model, adapted on demonstrations and online rollouts, to generate future video observations.
- Label generated trajectories with a reward model, then add them to RL training as synthetic transitions.
- Use Policy-Paced Learning: Q-aware selection balances positive and negative generated samples, and an entropy-based schedule reduces synthetic data when the actor is uncertain.
- Run video generation and world-model post-training asynchronously so robot control and policy updates keep running.

## Results
- On five real manipulation tasks, average success was 82% for WorldSample, compared with 56% for HIL-SERL, 69% for WMPO, and 64% for VLAW.
- Average training steps fell from 56K with HIL-SERL to 23K with WorldSample, a 59% reduction. Average training time fell from 83 min to 64 min.
- Per-task WorldSample success was 95% on Pushing, 95% on Insertion, 95% on Sorting, 84% on Pick & Place, and 42% on Assemble. HIL-SERL reached 84%, 63%, 66%, 55%, and 10% on the same tasks.
- World-model quality improved over demo-only adaptation: demo-only PSNR/SSIM/LPIPS was 10.27/0.428/0.809, while dual-view demo+rollout adaptation reached 29.89/0.925/0.035.
- On the insertion ablation, full WorldSample reached 95% success in 10K steps and 30 min. Without scheduling it reached 61% in 18K steps and 48 min; without Q-selection it reached 76% in 12K steps and 36 min; without full PPL it reached 86% in 32K steps and 69 min.

## Link
- [https://arxiv.org/abs/2607.02431v1](https://arxiv.org/abs/2607.02431v1)