Recoleta
GitHub
Source note

TTT-VLA: Test-Time Latent Prompt Optimization for Vision-Language-Action Models

Authors
Wenbo Zhang, Jianxiong Li, Shuai Yang, Sijin Chen, Jiajun Liu, Lingqiao Liu, et al.
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionTest Time TrainingLatent Prompt OptimizationRobot Foundation ModelSim2real
Open arXivMarkdown source

Summary

TTT-VLA adapts a frozen vision-language-action policy at deployment by optimizing learned latent prompt tokens with a self-supervised state-grounding loss. On SimplerEnv, it improves π0.5-based success rates across WidowX, Google Robot, and multi-embodiment settings.

Problem

  • VLA policies trained on large robot datasets still fail under deployment shifts in visuals, environment, or embodiment.
  • Prompt steering can improve behavior, but common prompt methods rely on human or external guidance and do not learn directly from deployment interaction.
  • RL post-training can learn from interaction, but it needs rewards and often updates the policy, which raises cost and stability issues.

Approach

  • The method adds a learned latent prompt z to the VLA conditioning input, alongside observation and explicit task context.
  • During training, the policy learns with the normal action loss plus a proxy state-grounding loss that predicts end-effector position and gripper state.
  • At test time, the robot collects interaction data in the current environment, freezes the policy backbone, and updates only the latent prompt with the state-grounding loss.
  • The implementation uses a Mixture-of-Transformers design with an action expert and a state-grounding expert, initialized from a pretrained π0.5 checkpoint.
  • In multi-embodiment training, the method assigns prompts by embodiment; in single-embodiment training, it uses prompt-action attention dropping and gradient restrictions to force the prompt to carry useful state-grounded context.

Results

  • On SimplerEnv WidowX single-embodiment tasks, mean success improves from 51.1% for π0.5 to 63.5% with state-grounded latent prompts, and to 67.4% after test-time prompt optimization.
  • On the same WidowX benchmark, the full method reports 67.4% mean success, above the strongest listed public baseline in the table, CogACT at 52.1%.
  • On Google Robot visual matching, mean success improves from 67.5% for π0.5 to 68.9% with latent prompt training, and to 72.4% with test-time training.
  • On Google Robot variant aggregation, mean success improves from 58.1% for π0.5 to 58.6% with latent prompt training, and to 60.1% with test-time training.
  • In the multi-embodiment setting trained on OXE-Aug Bridge V2 with nine embodiments and evaluated on WidowX, mean success improves from 22.8% for π0.5 to 28.5% with latent prompt training, and to 31.6% with test-time training.
  • Test-time training uses only prompt updates: 500 optimization steps for WidowX, 1000 for Google Robot, batch size 128, learning rate 1e-5, and 15–30 minutes on 8 NVIDIA H100 GPUs.

Link