Recoleta
GitHub
Source note

VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesis

Authors
Xiaolei Lang, Yang Wang, Yukun Zhou, Chaojun Ni, Kerui Li, Jiagang Zhu, et al.
Source
arXiv
Published
Collection
Embodied AI
Embodied Data SynthesisVision Language ActionWorld Action ModelRobot Data ScalingSim2real
Open arXivMarkdown source

Summary

VAG is a joint video-and-action generator for robot data synthesis. It aims to make synthetic embodied data usable for policy learning by generating future video frames and matching action trajectories together.

Problem

  • Robot foundation models need large teleoperation datasets, but collecting new robot demonstrations for each task or setting is expensive and slow.
  • Standard world models can generate future video, but they do not output the paired actions needed to train robot policies.
  • Two-stage methods that generate video first and then infer actions often lose video-action consistency and accumulate errors over long horizons.

Approach

  • VAG uses a dual-stream architecture with one branch for video generation and one branch for action generation, both conditioned on the first frame and a language instruction.
  • Both branches use flow matching and denoise in sync across the same generation steps, so video and action are produced together over a horizon of 93 frames, about 10 seconds at 10 Hz.
  • The video branch is built on Cosmos-Predict2 and predicts future video latents; the action branch is a modified 1D U-Net that predicts the action sequence.
  • At each denoising step, VAG takes the current clean video latent, compresses it with adaptive 3D pooling into a compact global embedding, and feeds that embedding to the action branch.
  • Training uses paired robot video-action trajectories plus text instructions extracted with Qwen2.5-VL and encoded with T5-XXL.

Results

  • On the AgiBot dataset, VAG beats SVD, Wan2.2, and Cosmos-Predict2 on several video metrics: FVD 965 vs 988 for CP2, 1152 for Wan2.2, and 1311 for SVD; LPIPS 0.320 vs 0.352, 0.325, and 0.421; PSNR 15.1 vs 14.2, 14.5, and 12.7. Its FID is 130, close to CP2 at 135 and Wan2.2 at 129.
  • On action generation for AgiBot, VAG reaches ED 0.81 and success rate 45%, compared with VAG-Video+AnyPos at ED 0.98 and 29%, and VAG-Video+ResNet at ED 1.54 and 8%.
  • On LIBERO action generation, VAG reaches ED 0.38 and success rate 79%, compared with VAG-Video+AnyPos at ED 0.55 and 66%, and VAG-Video+ResNet at ED 0.87 and 37%.
  • On LIBERO replay success, VAG gets 70% on Spatial, 72% on Object, 64% on Goal, 42% on Long, and 62% average. The AnyPos two-stage pipeline gets 54% average, and the ResNet pipeline gets 25% average.
  • For downstream VLA training on a self-collected real-robot dataset, the paper reports that VAG synthetic pretraining raises success rate from 35% to 55%, a gain of 20 percentage points.
  • The paper also claims the generated actions can be replayed on a real robot and can support executable trajectories, but the excerpt does not give fuller real-robot evaluation numbers beyond the 35% to 55% VLA improvement.

Link