Recoleta
GitHub
Source note

Semantic Anchoring for Robotic Action Representations

Authors
Yuan Xu, Youheng Shi, Chengyang Li, Wentao Zhu, Yizhou Wang
Source
arXiv
Published
Collection
Embodied AI
Robot Foundation ModelVision Language ActionGeneralist Robot PolicyRobot Data ScalingSim2real
Open arXivMarkdown source

Summary

Semantic Anchoring for Robotic Action Representations shows that action-only fine-tuning can erode the semantic structure inherited by VLA models, and that this erosion tracks out-of-distribution performance. Its training-time semantic anchoring method improves simulation and real-robot success while leaving the deployed inference model unchanged.

Problem

  • VLA models inherit semantic representations from pretrained vision-language encoders, but limited and narrow robot demonstrations can push action representations toward task-specific shortcuts.
  • This matters because semantic degradation is associated with weaker transfer to new instructions, objects, layouts, visual conditions, and task compositions, even when in-distribution success continues to rise.

Approach

  • Probe mid-layer action features against instruction embeddings across the full fine-tuning trajectory of (\pi_0), using bidirectional contrastive retrieval and task-disjoint LIBERO evaluation.
  • Add a training-time contrastive alignment loss that anchors action representations to a frozen semantic manifold from the EgoHOD text encoder.
  • Decompose action features into shared semantic and private execution-specific channels; align only the shared channel while reconstructing the original features and decorrelating the channels.
  • Discard all auxiliary alignment, decomposition, and reconstruction modules at inference, so the deployed policy has the same inference graph as the action-only baseline.

Results

  • On LIBERO, (\pi_0) improves from 89.3% to 92.4% average success, a +3.1 percentage-point gain; suite scores improve for Spatial (94.0% to 96.5%), Object (96.5% to 98.5%), Goal (90.0% to 92.5%), and Long (76.5% to 82.0%).
  • On SimplerEnv, the method improves (\pi_0) from 35.4% to 41.7% (+6.3 points) and SpatialVLA from 43.8% to 51.0% (+7.2 points).
  • On the real bimanual robot, average in-distribution success rises from 51.3% to 70.0% (+18.7 points), while out-of-distribution success rises from 49.5% to 71.0% (+21.5 points) over 20 trials per task condition.
  • Real-robot gains occur across all five OOD axes: language 75.0% to 85.0%, position 32.5% to 57.5%, object 52.5% to 77.5%, visual 57.5% to 75.0%, and task composition 30.0% to 60.0%.
  • During (\pi_0) fine-tuning, action-instruction alignment never returns to its pretrained level of 62.74%, while OOD success follows the alignment trend with Spearman (\rho=0.964); successful individual rollouts also show higher alignment than failed rollouts.
  • The excerpt reports ablations showing incremental gains from contrastive alignment and shared/private decomposition, with the strongest benefit at mid-network layer (k=10); it does not provide the exact ablation bar values.

Link