Source note
Abstract Sim2Real through Approximate Information States
Summary
This paper studies sim2real transfer when the simulator uses a coarser state than the real robot, so key dynamics are missing. It formalizes that setting as a state-abstraction problem and proposes ASTRA, a history-based simulator grounding method trained with a small amount of real-world data.
Problem
- The paper targets abstract sim2real: training an RL policy in a simplified simulator whose state omits real-world details, then transferring that policy to the real robot.
- This matters because high-fidelity robot simulators are expensive or impractical in large, complex domains, while abstract simulators are faster and easier to build.
- The main technical issue is that abstraction creates partial observability. Two different real states can map to the same abstract state, so standard sim2real methods that assume matched state spaces or only parameter mismatch can fail.
Approach
- The paper formalizes abstract sim2real using state abstraction in RL. The key claim is that grounding an abstract simulator requires using state-action history, because the abstract state alone is often not Markov.
- It introduces ASTRA (Augmented Simulation with self-predicTive abstRAction), which learns a recurrent latent state from abstracted real-world histories.
- ASTRA trains this latent state with three losses on paired simulator/real data: a latent transition loss to predict the next latent state, a reward prediction loss to keep task-relevant information, and an abstract next-state correction loss to adjust the simulator’s next-state prediction toward the real abstracted next state.
- The grounded simulator then rolls forward using the corrected abstract state and the learned latent dynamics, and an RL policy is trained on that grounded simulator. For deployment, a target-side encoder maps real-world histories into the same latent space.
Results
- The paper claims ASTRA enables successful transfer in both sim2sim and sim2real settings where direct transfer and several baselines fail.
- In AntMaze sim2sim navigation, ASTRA achieves the highest success rate on both U-Maze and Long Maze among the compared methods, including Direct Transfer, Domain Randomization, COMPASS, RMA, NAS, and DT+IQL. The excerpt does not provide the exact success-rate numbers from Figure 3.
- In a morphology-shift test on a modified Ant with 1.25× leg length, the ASTRA policy trained on the standard Ant reaches 65% success on U-Maze, versus 21% for direct transfer.
- For grounding data in AntMaze, ASTRA and NAS use 200 trajectories with average length 500 steps collected by a random behavior policy.
- The abstract-to-target gap in that setup is large: the target AntMaze state has 29 dimensions, while the abstract simulator uses only location and velocity.
- The excerpt states that ASTRA also works on real NAO humanoid tasks and simulated humanoid locomotion, but it does not include the final quantitative results for those experiments in the provided text.