Recoleta
GitHub
Source note

DexFuture: Hierarchical Future-State Visuomotor Targeting for Bimanual Dexterous Tool Use

Authors
Runfa Blark Li, Kuang-Ting Tu, Nikola Raicevic, Dwait Bhatt, Xinshuang Liu, Keito Suzuki, et al.
Source
arXiv
Published
Collection
Embodied AI
Dexterous ManipulationBimanual Tool UseVision Language ActionGeneralist Robot PolicyWorld Models
Open arXivMarkdown source

Summary

DexFuture predicts future hand-tool-object targets from egocentric RGB and robot state, then uses a target-conditioned dexterous policy to execute bimanual tool use. It aims to keep the control benefit of demonstration targets at deployment, where future demonstration states are unavailable.

Problem

  • Target-conditioned dexterous policies often need future targets taken from demonstrations, but a deployed robot cannot access future hand, tool, or object states.
  • Action-conditioned world-model planning over high-dimensional bimanual hand actions is too slow for stable contact-rich control.
  • The problem matters because tool use such as cutting, pouring, wiping, and shearing needs coordinated future guidance across two hands, the tool, and the object.

Approach

  • A high-level Future-State Visuomotor Target Predictor reads recent egocentric RGB frames, proprioception, and geometric cues.
  • The predictor builds structured tokens for hand links, the tool, and the object. Hand-link tokens come from projected link locations and local image features.
  • A horizon-conditioned transformer predicts sparse future targets at horizons {0, 2, 4, ..., 16}; intermediate targets are linearly interpolated during execution.
  • A low-level per-link transformer policy tracks the predicted 900-D target and outputs bimanual actions at control rate.
  • The predictor is trained with supervised future state and target losses, while the policy is trained with PPO using tracking rewards.

Results

  • On OakInk2 bimanual tool-use tasks, DexFuture reports 59.69% average success versus 66.52% for the privileged PhysGraph target baseline, reaching about 90% of oracle performance.
  • The no-target PhysGraph policy reaches about 7% average success in the abstract, showing that current-state feedback alone performs poorly on these tasks.
  • DexFuture runs at 60 Hz and is reported as about 250x faster than DexWM-style CEM planning with a future action-conditioned world model.
  • On fruit-knife cutting, DexFuture reaches 89.79% success versus 87.87% for privileged PhysGraph, with 0.61 cm tool/object translation error versus 0.98 cm.
  • On bread cutting, wiping, and shearing tasks, DexFuture remains close to privileged PhysGraph: 83.49% versus 90.05% on bread cutting, 56.96% versus 62.24% on big-brush wiping, and 30.69% versus 35.84% on one scissor-shearing task.
  • Future target prediction accuracy varies by task: examples include 0.87 cm 3D error and 99.78 PCK@10 on fruit-knife cutting, but 5.54 cm 3D error and 32.93 PCK@10 on one shearing task.

Link