Recoleta
GitHub
Source note

GIVE: Grounding Human Gestures in Vision-Language-Action Models

Authors
Pengfei Liu, Gen Li, Junqiao Fan, Boyu Ma, Jindou Jia, Yang Xiao, et al.
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionHuman Robot InteractionGesture GroundingRobot ManipulationFoundation Model Adaptation
Open arXivMarkdown source

Summary

GIVE adds human gesture understanding to pre-trained vision-language-action policies for robot handover tasks. It matters because text-only instruction following misses pointing and open-palm cues, which leads to wrong target selection and failed interaction.

Problem

  • Existing VLA policies usually treat manipulation as a text-driven task.
  • In human-robot interaction, gestures often carry the target object and task state.
  • Ambiguous or underspecified language causes intent grounding errors and execution failures.

Approach

  • GIVE injects gesture information without changing the base policy architecture.
  • It uses a visual path that overlays hand skeletons and, for pointing, a fingertip ray on the robot’s camera image.
  • It uses a semantic path that calls a pretrained VLM to turn stable gestures into a short text tuple for gesture type and execution instruction.
  • The semantic output is cached per phase, and object names are kept generic so the visual path handles target grounding.
  • The augmented image and text are fed into a pre-trained c0_0.5 VLA policy trained with a flow-matching action objective.

Results

  • On real-world grasp-then-handover trials with a Galaxea R1-Lite dual-arm robot, GIVE reaches 86.7% Identify SR, 80.0% Grasp SR, 80.0% React SR, and 80.0% Handover SR.
  • The baseline c0_0.5 gets 46.7% Identify SR, 6.7% Grasp SR, 3.3% React SR, and 0.0% Handover SR.
  • The paper reports a 40% gain in target object recognition accuracy and an 80% gain in overall task success rate over the baseline.
  • In the visual-semantic ablation, keypoints alone give 56.7% Identify SR, keypoints plus ray give 70.0%, and the full visual-plus-semantic system gives 86.7% Identify SR.
  • For semantic parsing, visual overlay raises target grounding accuracy from 40.0% to 90.0% over 20 trials.
  • On unseen participants, GIVE gets 8/10, 8/10, and 6/10 Identify SR for the seen participant, unseen participant A, and unseen participant B, while the baseline gets 4/10, 1/10, and 0/10.

Link