Recoleta
GitHub
Source note

DexSynRefine: Synthesizing and Refining Human-Object Interaction Motion for Physically Feasible Dexterous Robot Actions

Authors
Hyesung Lee, Hyunwoo Jung, Si-Hwan Heo, Sungwook Yang
Source
arXiv
Published
Collection
Embodied AI
Dexterous ManipulationHuman Object InteractionResidual RLSim To RealMotion Synthesis
Open arXivMarkdown source

Summary

DexSynRefine turns a small set of human-object interaction recordings into executable dexterous robot actions. It combines motion synthesis, residual reinforcement learning, and contact/dynamics adaptation for five manipulation tasks.

Problem

  • Dexterous teleoperation datasets are costly to collect, while human-object interaction recordings are easier to scale.
  • HOI data gives wrist, hand, and object motion, but not the contact forces and dynamics needed for robot execution; human and robot hands also differ in shape and actuation.
  • Sparse HOI demos often lack a matching reference for a new initial object pose, so raw retargeting fails outside the recorded cases.

Approach

  • The method collects 7 HOI demos per task across five tasks and expands them with object-centric SE(2) augmentation to about 300 trajectories per task, each with horizon T=220.
  • HOI-MMFP learns a latent motion manifold with a Transformer autoencoder, then uses conditional flow matching to generate wrist, hand-keypoint, and object trajectories from a task text embedding and the initial object pose.
  • A PPO teacher policy learns task-space residuals added to the generated wrist and fingertip targets; inverse kinematics converts these targets to 7-DoF arm and 16-DoF hand commands at 120 Hz.
  • A deployable student policy uses 30-step proprioceptive histories with a GRU contact estimator and a dynamics latent predictor to replace privileged simulation state.

Results

  • Under ±20 cm x/y and ±30° yaw initial-pose perturbations, HOI-MMFP gets first-frame wrist error of 0.015 m / 2.50°, compared with DiT-Full at 0.018 m / 2.65° and TC-VAE at 0.121 m / 12.88°.
  • HOI-MMFP has lower wrist jerk: 29.83±4.78 m/s^3 and 101.08±17.12 rad/s^3, compared with TC-VAE at 37.00±8.52 / 139.14±45.21 and DiT-Full at 41.03±8.23 / 189.52±33.82.
  • On Pick Up and Hammer, HOI-MMFP as the reference gives 52.6±3.04% simulated success, compared with DiT-Full at 49.10±4.12%; object translation error is 0.041 m vs 0.057 m, and object orientation error is 46.29° vs 51.42°.
  • Task-space residual actions give the best mean simulated success rate across five tasks: 68.1%. Per-task success rates are Pringles 71.5%, Watering Can 52.4%, Bowl 94.8%, Hammer 60.3%, and Book 61.6%; kinematic retargeting stays at 0.0-5.8%.
  • The full student policy beats adaptation ablations: on Hammer, 44.3% success vs 17.2% without contact and 7.5% without dynamics; on Watering Can, 51.8% vs 0.0% and 4.4%.
  • The abstract claims real-robot transfer on all five tasks, with a 50-70 percentage point gain over kinematic retargeting.

Link