Recoleta
GitHub
Source note

Action with Visual Primitives

Authors
Weilong Guo, Yuchen Wang, Renping Zhou, Yunfeng Zhang, Rui Fang, Yue Meng, et al.
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionRobot ManipulationVisual PrimitivesGeneralist Robot PolicySpatial Grounding
Open arXivMarkdown source

Summary

AVP is a VLA architecture that makes the VLM output visual target tokens before action prediction. It reports large real-robot gains over π₀.₅ on pick-and-place tasks that need precise spatial grounding.

Problem

  • Current VLA policies often map images and language straight to robot actions, so the action head must learn instruction parsing, spatial grounding, and motor control at the same time.
  • This matters because robot policies can fail when object layouts, target positions, or object appearances change, especially in dense scenes such as Chinese chess boards.
  • Prior visual-prompt methods use external detectors or VLM APIs, which add latency and can pass localization errors into the robot policy.

Approach

  • AVP splits the work: the VLM decides what target matters and where it is, then the action expert predicts how to move the robot.
  • The VLM autoregressively predicts visual primitives such as points, boxes, masks, or memory primitives for the next execution stage.
  • These primitives are projected into visual token space and fused with the observation tokens, then a flow-matching action expert predicts future robot actions.
  • The primitive labels come from end-effector kinematics through camera calibration, so the method avoids per-sample human prompt labels.
  • At inference, AVP uses only the observation, instruction, and robot state; it does not call an external detector, segmenter, or online VLM API.

Results

  • On Chinese chess manipulation, AVP reaches 90.28% average success versus 62.67% for π₀.₅, a +27.61 point gain. Metric breakdown: instruction following 98.61% vs 74.00%, pick 90.28% vs 72.00%, place 81.94% vs 42.00%.
  • On the same Chinese chess task, AVP runs at 0.27 s per instruction. Point-VLA with Kimi reports 48.15% average success and 37.32 s latency on a 20-instruction subset.
  • On domino placement, AVP reaches 88.19% average success versus 81.94% for π₀.₅. Pick improves from 87.50% to 100.00%, orientation from 93.75% to 100.00%, and place stays at 64.58%.
  • On general object pick-and-place, AVP reaches 86.18% average success versus 64.96% for π₀.₅. Pick improves from 71.79% to 90.24%, and place improves from 23.08% to 68.29%.
  • On unseen direct Chinese-chess transitions, AVP reports 83% average success with 100% instruction following, 90% pick, and 60% place, while π₀.₅ reports 0% on all four metrics.
  • In the visual-primitive ablation, average success rises from 78% with no primitive to 83% with boxes, 85% with box plus mask, and 91% with box plus mask plus memory.

Link