Source note
ViVa: A Video-Generative Value Model for Robot Reinforcement Learning
Robot Reinforcement LearningVideo GenerationValue FunctionVision Language ActionReal World Manipulation
Summary
ViVa uses a pretrained video diffusion model as a value function for robot reinforcement learning. It predicts future robot state and current task value from current images and proprioception, and the paper claims better value signals and better real-world box assembly inside the RECAP pipeline.
Problem
- Existing robot value models built on vision-language models read mostly static frames, so they miss temporal dynamics that matter for long-horizon manipulation under partial observability and delayed feedback.
- Weak value estimates hurt reinforcement learning because RECAP depends on value predictions for advantage estimation and policy improvement.
- The paper targets value estimation that tracks real task progress, catches execution errors, and transfers to novel objects.
Approach
- ViVa repurposes a pretrained video generator, Wan2.2, for value estimation instead of training a standard discriminative vision-language value model.
- Input: current multi-view RGB observations plus current robot proprioception. Output: a future proprioceptive state at horizon (K=50) and a scalar value for the current state.
- The model converts images, proprioception, and value into latent frames, then uses diffusion denoising with clean current observations as conditioning and noisy future-proprioception/value frames as targets.
- Value supervision uses a normalized return target derived from episode progress and final success or failure. Successful episodes map to values in ([0,1)); failed episodes shift to ([1,2)), giving a constant margin of 1.0 between success and failure at the same stage.
- ViVa replaces the VLM-based value function in RECAP while keeping the rest of the pipeline fixed, allowing a direct comparison of value-model design.
Results
- The excerpt gives no full quantitative benchmark table, so exact success-rate gains are not available here.
- The paper claims "substantial improvements" on real-world box assembly in both success rate and throughput when ViVa is integrated into RECAP, compared with prior value-model choices.
- Qualitative results on 3 real-world tasks—shirt folding, box packaging/assembly, and toilet paper organization—show that ViVa tracks task progress more smoothly than a VLM-based value model.
- In box assembly, ViVa shows sharp value drops at 2 highlighted failure events, while the VLM-based value stays mostly monotonic and misses those errors.
- In toilet paper organization, ViVa shows clear value increases at 2 highlighted milestones: roll alignment and label application; the VLM-based value stays mostly flat.
- Implementation details with concrete numbers: 3 tasks, single-epoch training, batch size 192, prediction horizon (K=50), loss weights (\lambda_{prop}=1.0) and (\lambda_{val}=0.5), 1 denoising step at inference, and training on 8 NVIDIA A800 GPUs.