Source note
FingerViP: Learning Real-World Dexterous Manipulation with Fingertip Visual Perception
Summary
FingerViP adds a small camera to each fingertip of a five-finger robot hand and trains a diffusion policy that uses those views together with a third-person camera. The paper targets dexterous manipulation in tight, occluded scenes where wrist-only or external views miss the contact area.
Problem
- Standard dexterous manipulation setups often rely on one wrist camera or distant external cameras, and both can lose sight of the object and finger contacts in cluttered or confined spaces.
- This matters for real-world dexterous tasks such as insertion, pressing, retrieval behind occlusions, and long-horizon hand-object interaction, where failure often comes from poor local perception near contact.
- Multi-finger hands need both scene-level context and close-range contact-relevant observations; prior camera placements give only part of that information.
Approach
- The system mounts an embedded miniature RGB camera in each fingertip of a 20-DoF robotic hand, giving five hand-centric views plus one third-view camera.
- A whole-body visuomotor policy takes six images, 26 joint angles, fingertip camera poses from forward kinematics, and per-finger joint currents, then predicts 26-DoF arm-hand actions.
- The policy is a transformer-based diffusion model trained from human teleoperation demonstrations. In simple terms, it learns to map recent observations to the next sequence of robot joint commands by denoising candidate actions.
- Image features come from a frozen CLIP ViT-B/16 encoder. Fingertip visual tokens are augmented with camera pose encodings to align moving viewpoints with robot state, and with finger current encodings to give contact cues such as loading or slip.
- The policy fuses fingertip views for local contact information and the third-view camera for global scene context.
Results
- Across four challenging real-world tasks, FingerViP reports an overall success rate of 80.8%.
- On confined-box button pressing, trained with 255 demonstrations of average length 7.2 s, FingerViP achieves 73.8% overall success over 42 evaluation rollouts and outperforms wrist-camera, third-view, fingertip-only, and mixed-camera baselines according to the paper.
- The hardware and data pipeline include 5 fingertip cameras, 1 third-view camera, a 20-DoF hand, and 26-DoF whole-body control with action prediction horizon n=16 and execution length m=8.
- Fingertip cameras run at 30 Hz with 640×480 resolution; teleoperation tracking runs at 60 Hz and robot control at 100 Hz; the reported average latency across five fingertip cameras is 6.7 ms.
- The abstract names four real-world evaluation tasks: pressing buttons inside a confined box, retrieving sticks from an unstable support, retrieving objects behind an occluding curtain, and long-horizon cabinet opening with object retrieval.
- The provided excerpt does not include the full task-by-task quantitative table beyond the 73.8% button-pressing result and the 80.8% overall success figure, so stronger per-baseline numeric comparisons are not available here.