Recoleta
GitHub
Source note

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

Authors
Guanxiong Chen, Qianjun Xia, Jiawei Peng, Heng Zhang, Bole Ma, Justin Qian, et al.
Source
arXiv
Published
Collection
Embodied AI
Real To SimWorld ModelingVision Language AgentsRobot ManipulationSimulator In The Loop
Open arXivMarkdown source

Summary

Agentic Real2Sim converts recorded robot-object interactions into physics-based, simulatable episode twins. It combines vision-language agents with deterministic perception, scene assembly, and simulator-in-the-loop refinement, supporting rigid manipulation and qualitative extensions to deformable and humanoid episodes.

Problem

  • Real-to-sim conversion requires recovering geometry, object states, physical parameters, camera and robot alignment, contacts, and trajectories; manual tuning makes this process slow and brittle.
  • The problem matters because reusable, real-world-aligned simulation episodes could support robot policy learning and evaluation across diverse interaction types.

Approach

  • A shared episode contract stores observations, actors, geometry, simulator states, physical and alignment parameters, the simulation backend, and replay metrics.
  • Four linked stages use agents for object discovery, keyframe and mask selection, physical-prior inference, scene preparation, and repair decisions, while deterministic tools perform segmentation, mesh recovery, depth estimation, pose tracking, calibration, and grasp optimization.
  • The system loads reconstructed episodes into MuJoCo and uses grasp sweeps or an agentic replay-refinement loop to adjust object placement based on simulation evidence.
  • Domain adapters reuse the contract for PhysTwin-style deformable interaction and BFM-Zero-style humanoid motion, but these extensions are evaluated qualitatively rather than with aggregate scores.

Results

  • On 100 randomly sampled DROID manipulation episodes, the Gemma 4 31B backend produced 48 replay successes, 8 partial outcomes, and 44 failures; success required a judge score of at least 8/10.
  • Replay-success counts across four VLM backends were 48/100 for Gemma 4 31B, 45/100 for Qwen 3.6 35B, 43/100 for GPT-5.4, and 37/100 for Claude Haiku 4.5.
  • Model bills ranged from $2.62 for Gemma 4 31B to $82.30 for GPT-5.4; Claude cost 3.5x, Qwen 5.0x, and GPT-5.4 31.4x as much as Gemma under the reported configuration.
  • Deformable-object and humanoid conversions show representative real-simulation matches and failure cases, but the paper reports no quantitative aggregate results for those domains.
  • The results support comparable observed replay outcomes from an open 31B VLM, while also showing that absolute replay success remained below 50% for every backend and remained sensitive to upstream perception and simulation errors.

Link