Source note
VLA-Trace: Diagnosing Vision-Language-Action Models through Representation and Behavior Tracing
Vision Language ActionRobot Policy DiagnosticsMechanistic InterpretabilityAttention KnockoutRepresentation Analysis
Summary
VLA-Trace is a diagnostic suite for vision-language-action policies that links representation drift, attention-based causal tests, and rollout behavior. It finds that π0.5 and OpenVLA use different routes for action decoding and both struggle with fine-grained language changes.
Problem
- VLA models can execute robot tasks, but their internal use of vision and language during action generation is hard to inspect.
- This matters because representation damage, weak language use, or visual shortcuts can cause policy failures that normal success rates do not explain.
Approach
- The method compares three checkpoints: the pretrained VLM, the pretrained VLA, and the task-finetuned VLA.
- It uses cross-modal CKA and checkpoint-drift CKA to measure how vision, text, and joint representations change during robot adaptation.
- It blocks selected attention paths during inference to test whether action tokens depend on image tokens, text tokens, or cross-modal prefill interaction.
- It runs behavioral probes with attention localization, visual patch masking, and input editing to test spatial grounding, shortcut use, and instruction following.
Results
- On LIBERO-10, π0.5 reaches 93.5% success in the all-paths baseline. Removing image access during generation drops it to 0.0%, while removing text access during generation leaves 39.0% success.
- On LIBERO Goal, Spatial, and Object, π0.5 keeps high success when generation text access is removed: 96.5%, 99.0%, and 98.0%. Removing generation image access gives 4.0%, 0.0%, and 0.0%.
- OpenVLA reaches 58.0% success on LIBERO-10 in the baseline. Removing generation text access drops it to 0.0% across LIBERO-10, Goal, Spatial, and Object; removing generation image access gives 1.0%, 16.0%, 44.0%, and 32.5%.
- Attention localization on LIBERO-10 shows action attention overlaps more with Robot+Object regions than object-only regions. For π0.5, full-rollout Robot+Object mass is 0.6328, IoU90 is 0.2233, and hit rate is 0.6349; for OpenVLA, the same metrics are 0.5882, 0.1965, and 0.9597.
- Visual masking reports strong dependence on visible task objects for π0.5. In one setup, its baseline success is 75.00% on LIBERO-10, 95.60% on LIBERO-Object, 95.80% on LIBERO-Spatial, and 80.00% on LIBERO-Goal; background replacement of target objects drops these by an average of 76.45 points.
- The main qualitative claim is that both models generate visually grounded trajectories, but both have limited fine-grained semantic following when prompts or inputs are edited.