Recoleta
GitHub
Source note

TacCoRL: Integrating Tactile Feedback into VLA via Simulation

Authors
Siyu Ma, Yuqi Liang, Chang Yu, Yunuo Chen, Hao Su, Yixin Zhu, et al.
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionTactile FeedbackSim2realRobot RLContact Rich Manipulation
Open arXivMarkdown source

Summary

TacCoRL adds tactile sensing to pretrained VLA robot policies and trains contact correction in simulation before real deployment. It targets bimanual contact-rich tasks where cameras miss local alignment and pressure cues.

Problem

  • Contact-rich manipulation tasks such as insertion, assembly, and puzzle placement need local contact state, but vision often cannot see misalignment, blockage, pressure, or contact location.
  • Real robot demonstrations are expensive and mostly show successful nominal behavior, so they give weak supervision for near-failure contact states.
  • Collecting many off-nominal contact rollouts on hardware can damage sensors and slows training because resets are costly.

Approach

  • The policy starts from a pretrained VLA backbone and adds tactile tokens from a recent tactile history window.
  • A binary contact gate removes tactile tokens when readings look like background noise, so touch affects the policy mainly during contact.
  • Sim-real co-training mixes real demonstrations with simulated teleoperation and MimicGen trajectories to warm-start tactile-conditioned actions.
  • PPO reinforcement learning runs in a real-aligned simulator with sparse task rewards, while a supervised loss on real trajectories keeps the policy close to real robot observations and actions.
  • The final policy deploys directly on the real robot without privileged simulator state or online real-world RL.

Results

  • Across 4 real-world bimanual contact-rich tasks, the final visuo-tactile policy reaches 72.5% average success, compared with 50.0% for the RL post-trained vision-only policy.
  • Real-world task success after RL post-training is 70% on Test Tube Insertion, 45% on Do Puzzle, 95% on Assembly #1, and 80% on Assembly #2 for the visuo-tactile policy.
  • On the same real tasks, vision-only RL post-training reaches 35%, 25%, 80%, and 60%, respectively.
  • In simulation, RL with co-training raises average success to 78.5% for visuo-tactile policies and 60.5% for vision-only policies.
  • Direct sparse-reward RL from the base VLA gets 0.0 success across all 4 simulated tasks, showing that co-training is needed before RL refinement.
  • The ablation on Assembly #2 reports that real-data anchor weights β=0.1 and β=1.0 raise real-world success to 80%, while no anchor gives 45% under the same α=0.5 co-training ratio.

Link