Source note
PokeVLA: Empowering Pocket-Sized Vision-Language-Action Model with Comprehensive World Knowledge Guidance
Vision Language ActionRobot ManipulationEmbodied Foundation ModelSpatial ReasoningMulti View Learning
Summary
PokeVLA is a small vision-language-action model for robot manipulation that adds embodied world knowledge before action training. It targets better spatial understanding, target awareness, and robustness than lightweight VLA baselines.
Problem
- Existing VLA models often pass generic vision-language features into an action head with weak alignment to robot manipulation, which can make learning inefficient and costly.
- Pre-trained VLM knowledge does not match robot tasks well enough, especially for spatial relations, multi-view consistency, and target-focused guidance.
- These gaps matter because robot manipulation needs accurate object grounding, spatial reasoning, and stable behavior under scene changes and perturbations.
Approach
- The method has two stages. First, it pre-trains a compact vision-language model called PokeVLM on about 2.4M to 2.5M multimodal samples covering general VQA, spatial grounding, affordance learning, and embodied reasoning.
- PokeVLM uses a Qwen2.5-0.5B language model with SigLIP and DINOv2 visual encoders, so the base model stays small while adding embodied visual-language knowledge.
- During VLA post-training, the model learns a special
<SEG>token that predicts manipulation-target segmentation masks from both base and wrist camera views. This gives the policy a shared representation of the target across views. - It aligns visual hidden states with features from the 3D geometry model VGGT during training, so the model learns scene structure without needing the geometry model at inference time.
- An action head with learnable action queries gathers language, vision, segmentation, geometry, and robot-state features and feeds them into the action expert for action prediction.
Results
- On LIBERO-Plus, when trained on the LIBERO-Plus dataset, PokeVLA beats OpenVLA-OFT by 4.0% and VLA-Adapter by 2.5% in total success rate.
- In a transfer setting trained only on original LIBERO and tested on LIBERO-Plus variations and perturbations, PokeVLA exceeds OpenVLA-OFT by 9.7% and VLA-Adapter by 20.2% in average success rate.
- In real-world tasks with spatial and color references, it reports a 12.5% success-rate gain over similar-scale baselines.
- Under real-world perturbations, the reported gain grows to 20.0%, which the paper presents as evidence of stronger robustness.
- The pre-training corpus contains roughly 2.4M to 2.5M samples, including 665K general, 694K grounding, 553K affordance, and 511K reasoning examples.