Source note
Learning When to See and When to Feel: Adaptive Vision-Torque Fusion for Contact-Aware Manipulation
Summary
This paper studies how to combine vision and joint torque signals for contact-rich robot manipulation with diffusion policies. Its main claim is that torque should be ignored during free motion and added only during contact through an adaptive fusion rule.
Problem
- Vision-only manipulation policies miss contact state, alignment quality, friction, and insertion progress in tasks where visual change is small or the scene is occluded.
- Naive vision-plus-torque fusion can hurt performance because joint torque signals during free-space motion contain noise and inertia effects.
- Prior work proposes auxiliary losses, routing, and gating, but this paper argues there was no clear head-to-head comparison inside the same diffusion-policy setup.
Approach
- The policy uses a diffusion-policy backbone with two RGB views, joint positions, and a 10-step history of 7 joint external torques from a Franka Research 3 robot.
- A contact gate checks whether any joint external torque passes a threshold. If there is no contact, the torque feature is replaced by a learned default vector; if there is contact, the real torque feature is used.
- The model trains two separate U-Nets: one for vision plus proprioception, and one for gated torque plus proprioception.
- At each diffusion denoising step, the final noise prediction is a learned blend of the two experts. A small MLP predicts the torque weight from image and gated torque features, and the weight is forced to zero when there is no contact.
- The paper benchmarks this method against vision-only, feature concatenation, torque gating alone, auxiliary future-torque prediction, and mixture-of-experts fusion.
Results
- Across three real-world tasks, the proposed method reaches 82.0% average success: bottle placement 16/20, connector pull-out 7/10, egg-boiler lid opening 18/20.
- The strongest baseline in Table I is torque gating at 68.0% average success: 14/20, 5/10, 15/20. The reported gain is 14 percentage points over this baseline.
- Vision-only gets 30.0% average success (8/20, 0/10, 7/20). Feature concatenation also gets 30.0% (3/20, 0/10, 12/20). Auxiliary goals get 28.0% (6/20, 1/10, 7/20). MoE gets 24.0% (5/20, 1/10, 7/20).
- On the weight-based bottle task, the proposed method gets 9/10 for empty bottles and 7/10 for full bottles, for 16/20 total. Vision-only gets 8/10 on empty and 0/10 on full, which shows it fails to infer weight.
- On egg-boiler lid opening single-attempt performance, torque gating gets 20.0% first-try success with 110.7 average task-horizon steps, while the proposed method gets 60.0% with 87.1 steps.
- The paper’s strongest qualitative claim is that most of the gain comes from filtering torque during free motion, while the adaptive guidance weight improves precision once contact starts.