Recoleta
GitHub
Source note

ProGAL-VLA: Grounded Alignment through Prospective Reasoning in Vision-Language-Action Models

Authors
Nastaran Darabi, Amit Ranjan Trivedi
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionRobot GroundingAmbiguity DetectionGeneralist Robot Policy3d Scene Representation
Open arXivMarkdown source

Summary

ProGAL-VLA is a vision-language-action model that forces the robot to verify which 3D scene entity an instruction refers to before acting. The paper targets a common VLA failure mode where the policy ignores language and follows visual shortcuts.

Problem

  • Existing VLA systems often show language ignorance: changing the instruction does not change behavior enough, because the policy leans on visual priors instead of instruction meaning.
  • This matters for robot manipulation because a robot can pick the wrong object, fail on spatial or relational commands, or become unstable when the scene or camera changes.
  • Standard multimodal fusion does not check whether the intended symbolic goal actually matches a real, actionable entity in the current 3D scene.

Approach

  • ProGAL-VLA splits the system into a slow planner and a fast controller. The planner maps the instruction and observation to a short symbolic sub-goal such as grasp_red_block.
  • A Grounded State Module (GSM) builds an entity-centric 3D scene representation with tracked objects and short-term memory, so the model reasons over objects rather than raw image patches.
  • A State Alignment Cross Attention (SACA) module matches the symbolic sub-goal to the 3D entities and outputs a verified goal embedding g_t. The action policy receives this verified embedding, not raw language.
  • Training adds a Grounding Alignment Contrastive (GAC) loss, an InfoNCE-style objective that pulls the symbolic goal toward the correct entity embedding and pushes it away from wrong entities.
  • The attention entropy over entities is used as an ambiguity score. High entropy means the instruction does not identify one object clearly, so the model can abstain and ask for clarification.

Results

  • On LIBERO-Plus robustness, ProGAL-VLA reports 85.5 total versus 79.6 for OpenVLA-OFT+ and 17.3 for base OpenVLA.
  • Under robot perturbations on LIBERO-Plus, performance rises from 30.3 to 71.5. Other reported category scores include camera 93.2 vs 92.8, language 93.6 vs 85.8, and layout 86.7 vs 77.6 compared with OpenVLA-OFT+.
  • The paper claims 3x-4x lower language ignorance across simple, spatial, and relational instructions.
  • Grounded entity retrieval improves from 0.41 to 0.71 Recall@1 with N=8 candidates.
  • On the Custom Ambiguity Benchmark, ambiguity detection reaches AUROC 0.81 versus 0.52 and AUPR 0.79.
  • On ambiguous inputs, clarification behavior rises from 0.09 to 0.81 without reducing success on unambiguous instructions, according to the paper.

Link