Source note
See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models
Summary
Robot-centric pointmaps give VLAs dense 3D scene coordinates in the same frame used for robot actions. This reduces viewpoint-related frame mismatch while reusing pretrained 2D visual encoders.
Problem
- VLAs usually observe scenes in camera coordinates but predict actions in the robot frame, forcing the policy to learn the camera-to-robot transformation.
- The problem grows when demonstrations use varied camera placements or deployment uses an unseen viewpoint.
- Camera-aware inputs, depth maps, and point clouds either leave the transformation to the policy or discard the image grid used by pretrained VLAs.
Approach
- Lift each RGB-D pixel into 3D, transform it from the camera frame into the robot base frame, and subtract the current end-effector position.
- Store the resulting XYZ coordinates in an image-shaped pointmap with the same H×W layout as the RGB image.
- Encode the pointmap with a visual tower initialized from the RGB encoder, then add its tokens element-wise to the RGB tokens.
- This adds robot-frame metric geometry without a point-cloud encoder, voxelization, extra token sequence, or inference-time rendering stage.
Results
- On RoboCasa, the 24-task average success rate increased from 55.3% to 62.9% for π₀.₅, a gain of 7.6 points, and from 37.2% to 41.4% for SmolVLA, a gain of 4.2 points.
- The π₀.₅ pointmap model scored 62.9%, above KYC at 59.1%, PointVLA at 57.3%, and GeoVLA at 57.1%.
- In controlled RoboCasa studies, RGB + pointmap reached 34.7% versus 31.6% for RGB plus Plücker rays and depth; end-effector centering raised this to 36.9%.
- Under increasing training-camera variation, RGB success fell 9.6 points, from 34.5% to 24.9%, while the pointmap model fell 1.8 points, from 37.6% to 35.8%.
- In real-robot Franka experiments, the gain over RGB increased from 5.0 points at a training camera placement to 11.7 points at an unseen camera placement.