Source note
VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Models
Summary
VEGA improves OpenVLA-OFT by adding 3D-aware visual feature alignment during training, then removing the extra teacher and projector at inference.
Problem
- VLA visual encoders are trained mainly on 2D image data, so they can miss depth, relative position, object height, and viewpoint cues needed for precise manipulation.
- Existing implicit spatial grounding methods align LLM-level visual tokens and require layer search; the alignment target depends on the task and mixes geometry with language context.
Approach
- Use frozen DINOv2-FiT3D as a spatial teacher. FiT3D is DINOv2 tuned with multi-view 3D Gaussian Splatting supervision.
- Align OpenVLA-OFT's DINOv2 visual encoder output to DINOv2-FiT3D patch features through a LayerNorm plus two-layer MLP projector.
- Train with standard action prediction loss plus cosine feature alignment loss, with λ = 0.1 in the main experiments.
- Discard the teacher and projector at inference, so runtime is the same as the base VLA.
Results
- On RoboTwin 2.0, across 6 bimanual tasks and 100 trials per task, VEGA reports 67.5% average success on Easy and 30.7% on Hard.
- It beats OFT + Spatial Forcing by 3.3 percentage points on Easy (67.5% vs 64.2%) and 2.9 points on Hard (30.7% vs 27.8%).
- Compared with OpenVLA-OFT, VEGA improves the average from 56.0% to 67.5% on Easy and from 22.7% to 30.7% on Hard.
- Task-level gains include Move Card Away Hard at 43% vs 34% for OpenVLA-OFT, Click Bell Hard at 53% vs 46%, and Place Shoe Hard at 25% vs 9%.
- Training efficiency claim: on Move Playingcard Away Easy, VEGA at 10k steps reaches about the same success rate as OpenVLA-OFT at 60k steps.
- Data efficiency claim: with 25% of demonstrations on Move Playingcard Away Easy, VEGA is 10 percentage points above OpenVLA-OFT.