Source note
Learning Versatile Humanoid Manipulation with Touch Dreaming
Summary
This paper presents a real-world humanoid manipulation system that combines stable whole-body control, dexterous hands, and tactile sensing. Its main model, Humanoid Transformer with Touch Dreaming (HTD), adds future-touch prediction to behavior cloning and reports large gains on contact-rich tasks.
Problem
- Real humanoid loco-manipulation is hard because the robot must keep whole-body balance, move dexterous hands, and react to changing contact at the same time.
- Vision and proprioception alone miss parts of contact state, which hurts performance on insertion, deformable objects, tool use, and bimanual transport.
- Prior humanoid systems usually miss at least one of these pieces: whole-body control, full dexterous-hand control, tactile sensing, or tactile modeling.
Approach
- The system has an RL-trained lower-body controller for stable locomotion and torso tracking, plus VR teleoperation, upper-body IK, and dexterous hand retargeting for data collection and execution.
- The policy model, HTD, is a multimodal encoder-decoder Transformer that takes multi-view RGB, proprioception, hand-joint force signals, and tactile inputs.
- Training uses single-stage behavioral cloning with auxiliary "touch dreaming" losses. Along with predicting action chunks, the model predicts future hand-joint forces and future tactile latents.
- Future tactile targets come from an EMA target encoder, so the model learns a latent prediction target without a separate tactile pretraining stage or an inference-time world model.
- Tactile input is encoded by hand region and finger region, then fused with the other modalities in the shared Transformer trunk.
Results
- On 5 real-world contact-rich humanoid tasks, HTD reports a 90.9% relative improvement in average success rate over the stronger ACT baseline.
- The task set includes Insert-T, Book Organization, Towel Folding, Cat Litter Scooping, and Tea Serving.
- The paper states latent-space tactile prediction beats raw tactile prediction by a 30% relative gain in success rate in ablations.
- The system handles a tight-tolerance insertion task with 3.5 mm clearance.
- Deployment runs the learned policy at 30 Hz, while the lower-body controller, IK solver, and hand retargeter run at 50 Hz.
- The excerpt does not provide per-task success rates, dataset size, or absolute baseline numbers, so the strongest quantitative claims available are the 90.9% average relative gain and the 30% ablation gain.