Recoleta
GitHub
Source note

Test-Time Training for Visual Foresight Vision-Language-Action Models

Authors
Sangwu Park, Wonjoong Kim, Yeonjun In, Sein Kim, Hongseok Kang, Chanyoung Park
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionTest Time TrainingVisual ForesightOut Of DistributionRobot Manipulation
Open arXivMarkdown source

Summary

T³VF adds test-time training to Visual Foresight VLA models so they can adapt when visual conditions shift at deployment. It uses the gap between a predicted future image and the later observed image as supervision, then filters updates with action variance.

Problem

  • Visual Foresight VLA models first predict a future visual state, then generate actions from that prediction, so an out-of-distribution visual shift can damage both image prediction and action generation.
  • The paper targets LIBERO-Plus perturbations such as robot state, language, noise, layout, background, camera, and light changes.
  • This matters because VF-VLA success can drop under deployment shifts even when the base model works well on in-distribution LIBERO tasks.

Approach

  • At step t, the model predicts a future observation ô_{t+n} and executes an action. After n steps, the environment gives the actual observation o_{t+n}.
  • T³VF treats (ô_{t+n}, o_{t+n}) as a self-supervised training pair and updates the image-prediction path with the same image loss used in training.
  • The update changes only the learnable query tokens q; the VLM backbone, image head, and action head stay frozen.
  • To avoid noisy updates, the method samples K=5 actions, computes their squared L2 variance, and accepts an update only when the variance is in the lower ρ=0.3 quantile of a recent buffer.
  • The reported hyperparameters are prediction gap n=4, batch size B=4, variance buffer size 10, action samples K=5, and percentile threshold ρ=0.3.

Results

  • On LIBERO-Plus with perturbed training, Mantis + T³VF reaches 52.1% average success versus 49.3% for Mantis, a +2.8 percentage-point gain, about +5.7% relative.
  • In the same setting, the largest listed gains are Camera 55.3% vs 50.5% (+4.8 points), Light 72.4% vs 67.8% (+4.6 points), Background 63.0% vs 60.3% (+2.7 points), and Layout 44.9% vs 42.3% (+2.6 points).
  • Without perturbed training, T³VF gives a smaller gain: 40.3% average success versus 39.8% for Mantis, or +0.5 points.
  • The Robot-perturbation ablation reports 29.0% for Mantis, 29.8% with unfiltered test-time training, 28.6% with a fixed variance threshold, and 31.8% with the full adaptive buffer.
  • Runtime on the Robot perturbation is about 1.3× the base per-episode time for T³VF, compared with about 1.7× for unfiltered test-time training.

Link