Source note
Tune to Learn: How Controller Gains Shape Robot Policy Learning
Summary
This paper studies how low-level PD controller gains change robot policy learning. Its main claim is that gains should be chosen for learnability within a training setup, not by assuming they directly set task compliance.
Problem
- Learned manipulation policies often run through position controllers, but gain selection is still treated as a classical control choice based on stiffness or compliance.
- That view misses how reactive policies work: the policy and controller together create the closed-loop behavior, so gains mainly change how easy the policy is to train and transfer.
- This matters for behavior cloning, reinforcement learning, and sim-to-real transfer because the same gain setting can help one pipeline and hurt another.
Approach
- The paper reframes controller gains as an interface parameter and tests their effect across three settings: behavior cloning, RL from scratch, and zero-shot sim-to-real transfer.
- For behavior cloning, it builds controlled datasets with Torque-to-Position Retargeting (TPR) so trajectories stay close across gain settings while the position-command labels change with the gains.
- TPR converts torque demonstrations into position targets using the PD control equation, then replays them at policy rate to isolate the effect of gain-dependent action labels.
- For RL, it re-tunes action-space and training hyperparameters for each gain setting so poor results are less likely to come from mismatched hyperparameters.
- For sim-to-real, it performs gain-specific system identification, trains RL policies in the matched simulator, and evaluates zero-shot transfer on a real Franka robot, with domain-randomization and control-frequency ablations.
Results
- Behavior cloning: Across several manipulation tasks and embodiments, the best closed-loop success concentrates in the compliant, overdamped gain regime; stiff or weakly damped gains reduce success. The excerpt does not provide exact per-task success numbers from the heatmaps.
- TPR fidelity: Retargeted trajectories keep >=90% success and joint-position MSE < 1e-3 across gain settings up to 25x decimation (20 Hz).
- Teleoperation study: The user study includes 12 users, 1-hour sessions, and 1,297 total trials on a box-pushing task. The excerpt states that results are reported later, but it does not include the numeric outcome table here.
- Reinforcement learning: The paper claims RL can reach successful policies in all tested gain regimes when hyperparameters are tuned to match the gain setting, across manipulation and locomotion tasks. The excerpt does not include final success percentages or reward numbers.
- Sim-to-real: The paper claims stiff and overdamped gains worsen motor-level sim-to-real transfer, while transfer is evaluated over 30 real-world rollouts per gain setting using state-trajectory MSE. The excerpt does not include the final MSE values.
- Data practice observation: By analyzing DROID and Open X-Embodiment datasets, the authors report that existing datasets appear to use stiff tracking behavior as a default, based on command-following curves rather than published gain values.