Recoleta
GitHub
Source note

Learning while Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies

Authors
Yi Wang, Xinchen Li, Pengwei Xie, Pu Yang, Buqing Nie, Yunuo Cai, et al.
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionGeneralist Robot PolicyFleet Scale RLRobot Data ScalingOffline To Online RL
Open arXivMarkdown source

Summary

LWD trains a single VLA robot policy during deployment by mixing offline robot data with new fleet rollouts. On 16 dual-arm robots and 8 real-world manipulation tasks, it reports 95% average success after a few hours of online interaction.

Problem

  • Offline VLA pretraining misses distribution shifts, long-tail failures, task variants, and human corrections seen after deployment.
  • Pure imitation learning cannot use failed autonomous trials, partial progress, sparse rewards, and recoveries as RL signals.
  • Existing robot RL often improves one short-horizon or task-specific policy, so it does not solve continual post-training for a shared generalist policy.

Approach

  • Start with a pretrained flow-based Vision-Language-Action policy that outputs action chunks from observations and language instructions.
  • Deploy current checkpoints to a shared fleet, collect autonomous rollouts plus optional human interventions, add them to an online replay buffer, and retrain on mixed offline and online replay.
  • Use Distributional Implicit Value Learning: fit a distribution over replay action-values, then use a high quantile as the TD bootstrap target instead of a scalar expectile value.
  • Adapt the quantile level with value-distribution entropy, using lower optimism when the value distribution is diffuse.
  • Extract the policy with Q-learning via Adjoint Matching, which turns critic action gradients into local training targets for the flow action generator.

Results

  • The real-world evaluation uses 16 dual-arm robots across 8 manipulation tasks.
  • The task set includes semantic grocery restocking and long-horizon tasks such as Gongfu tea, cocktails, and fruit juice, with reported execution times of 3–5 minutes for the long-horizon tasks.
  • A single generalist policy reaches an average success rate of 95% across all tasks.
  • The paper says LWD improves over the pretrained VLA and beats relevant baselines by large margins, but the excerpt does not provide baseline success rates.
  • The largest gains are reported on long-horizon tasks, where sparse rewards can be propagated across partial progress with dynamic programming.
  • The online improvement phase is reported to need only a few hours of real-world interaction.

Link