Source note
LaST-HD: Learning Latent Physical Reasoning from Scalable Human Data for Robot Manipulation
Vision Language ActionHuman To Robot TransferLatent World ModelsRobot ManipulationDexterous Manipulation
Summary
LaST-HD trains a VLA robot policy to learn from human-hand demonstrations by aligning human and robot trajectories in a shared latent dynamics space. It pairs this with a low-cost motion-capture glove and a mixed human-robot training recipe for manipulation across grippers and dexterous hands.
Problem
- Robot VLA policies need many real-robot demonstrations, but teleoperation is slow, hardware-bound, and costly.
- Human-hand demonstrations are easier to collect, but direct transfer fails when human hands and robot grippers or dexterous hands differ in shape, joints, and dynamics.
- The paper targets cross-embodiment manipulation learning where human data can improve robot generalization to new objects, positions, and scenes.
Approach
- LaST-HD uses a Mixture-of-Transformers VLA model built on Janus-Pro, SigLIP-Large, and a 1.5B DeepSeek-LLM backbone, with one expert for latent reasoning and one expert for action generation.
- An action-conditioned world model is trained on unpaired human-hand and robot trajectories. Its forward-dynamics features become latent targets for the VLA reasoning expert.
- The policy learns robot actions with flow matching while its latent tokens are supervised by cosine similarity to the world-model targets.
- The OOL Glove records 21 hand-wrist keypoints and turns human motions into supervision for grippers and dexterous hands through fingertip-distance rules and inverse-kinematics retargeting.
- Training uses mixed human-robot co-training, then human-hand online correction at failure states while replaying older data to reduce forgetting.
Results
- In-domain evaluation covers 6 real-world tasks across 3 embodiments. LaST-HD averages 0.73 success, compared with 0.62 for π0.5, 0.52 for Cosmos-Policy, and 0.63 for LaST0.
- On in-domain tasks, LaST-HD reaches 0.95 on Sort Fruits and 0.80 on Put Items to Bag and Zip. The mixed human-robot variant averages 0.68 using 50 robot demonstrations and 50 OOL Glove demonstrations per task.
- In generalization with extra human data only, LaST-HD reaches a 0.56 global average, compared with 0.46 for LaST0 with the same unseen human-data setting.
- By scenario, LaST-HD with unseen human data averages 0.41 on unseen positions, 0.58 on unseen objects, and 0.68 on unseen backgrounds across the 6 tasks.
- OOL Glove is reported as under 100 g per glove, over 200 Hz, under 10 ms latency, with sub-millimeter average RMS keypoint position error, and 4-5× faster data collection than robot teleoperation.
- Human-hand online correction is reported to push novel-environment accuracy above 90% using 20 minutes of OOL Glove data; in the shown Sort Fruits correction study, 20 trajectories reach 100% on unseen background and 60 trajectories reach 100% on unseen object.