--- kind: ideas granularity: day period_start: '2026-07-21T00:00:00' period_end: '2026-07-22T00:00:00' run_id: 9263b2f3-904a-4e7c-abe4-3407a1f0ee6c status: succeeded topics: - embodied world models - robotics - action representations - real-to-sim - interactive simulation tags: - recoleta/ideas - topic/embodied-world-models - topic/robotics - topic/action-representations - topic/real-to-sim - topic/interactive-simulation language_code: en pass_output_id: 373 pass_kind: trend_ideas upstream_pass_output_id: 372 upstream_pass_kind: trend_synthesis --- # Counterfactual and trace-based tests for robot world models ## Summary Replayable episode twins can supply counterfactual supervision that recorded robot videos lack, while spatial traces can turn whole-episode replay failures into repairable alignment, contact, and dynamics errors. Bidirectional visual action models also need cycle tests that verify whether inferred robot motion actually produces the requested object outcome. ## Counterfactual simulator reruns for separating passive dynamics from robot effects Robot-learning teams can use successful real-to-sim conversions to create paired transitions from the same initial state: replay the recorded action, replace it with a null action, and perturb its timing or direction. Agentic Real2Sim already reconstructs geometry, object state, physical parameters, and trajectories as runnable episodes; DWM shows why the resulting counterfactuals matter, reporting a mean 13.1-point planning-success improvement when training separates persistent world effects from action-driven change on simulated benchmarks. This combination offers a practical route to test that decomposition on realistic interaction records without collecting matched counterfactuals on physical robots. The cheapest check is to rerun the successfully reconstructed DROID episodes under null and perturbed actions, train with and without the paired supervision, and evaluate on held-out recordings containing sliding, rebound, or post-contact motion. Because only 48 of 100 episodes replayed successfully with Agentic Real2Sim’s best tested backend, results should also be stratified by reconstruction quality; otherwise simulator error could be mistaken for a learned world effect. ### Sources - [DWM: Separating World Effects from Actions in Latent World Models](../Inbox/2026-07-21--dwm-separating-world-effects-from-actions-in-latent-world-models.md): DWM separates action-invariant and action-driven transition components and reports a 13.1 percentage-point mean absolute CEM planning gain across three persistent-dynamics benchmarks. - [Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents](../Inbox/2026-07-21--agentic-real2sim-physics-based-world-modeling-with-vision-language-agents.md): Agentic Real2Sim stores physical episode state for simulator replay, but its best tested backend successfully replayed only 48 of 100 DROID episodes. ## Trace-level replay diagnostics for automated real-to-sim repair Real-to-sim engineers should add object masks, gripper traces, contact points, and placement targets to each episode’s replay record, then repair the first structured mismatch instead of relying mainly on an end-to-end replay score. Agentic Real2Sim exposes failures from perception, alignment, scene assembly, and physics through one simulator-in-the-loop workflow. RoboInter1.5 shows that these intermediate signals can be annotated at scale, while Masked Visual Actions demonstrates that pixel-space entity trajectories can condition scene-response prediction across embodiments. Together, they suggest a diagnostic that distinguishes an incorrectly aligned robot from a wrong contact event or object trajectory before another full replay is attempted. A low-cost evaluation can reuse Agentic Real2Sim’s DROID-100 set: measure per-frame robot-mask, object-track, contact-time, and final-placement errors, then test whether those residuals predict the existing replay judgments and help an automated repair step choose the parameter family to revise. The useful outcome is not merely a higher visual score, but fewer failed repair iterations and better physical replay. ### Sources - [Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents](../Inbox/2026-07-21--agentic-real2sim-physics-based-world-modeling-with-vision-language-agents.md): The conversion pipeline combines deterministic perception and simulation refinement, yet all tested backends remained below 50% full replay success on DROID-100. - [RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation](../Inbox/2026-07-21--robointer1-5-a-holistic-intermediate-representation-suite-for-embodied-world-modeling-and-robotic-manipulation.md): RoboInter1.5 provides dense object and gripper grounding, affordances, contact points, placement proposals, and motion traces across more than 230,000 episodes. - [Masked Visual Actions for Unified World Modeling](../Inbox/2026-07-21--masked-visual-actions-for-unified-world-modeling.md): Masked Visual Actions uses pixel-space entity trajectories and achieved LPIPS 0.0945 on DROID versus 0.362 for Ctrl-World. ## Forward–inverse cycle tests for visual robot action interfaces Benchmark maintainers can test visual action interfaces as a closed loop: infer robot motion from a desired object trajectory, feed that inferred motion back into the forward model, and score whether the predicted object reaches the requested state with the correct contact and placement. Masked Visual Actions uses one checkpoint in both directions, but image fidelity alone does not show that the inferred action and predicted consequence agree. RoboInter1.5 supplies the object, gripper, contact, affordance, and placement representations needed to score that agreement across varied manipulation episodes and robot arms. The first test can mask robot motion in held-out clips, infer it from the observed object path, run the inferred path forward, and compare endpoint, contact timing, placement validity, and conventional video metrics with the recorded outcome. Reporting cycle errors by robot embodiment would reveal whether a visually shared interface is genuinely embodiment-flexible or only produces plausible-looking motion. ### Sources - [Masked Visual Actions for Unified World Modeling](../Inbox/2026-07-21--masked-visual-actions-for-unified-world-modeling.md): Masked Visual Actions represents action as a partially revealed entity trajectory and uses the same checkpoint for forward scene prediction and inverse robot-motion synthesis. - [RoboInter1.5: A Holistic Intermediate Representation Suite for Embodied World Modeling and Robotic Manipulation](../Inbox/2026-07-21--robointer1-5-a-holistic-intermediate-representation-suite-for-embodied-world-modeling-and-robotic-manipulation.md): RoboInter1.5 spans 571 scenes and six robot-arm types with dense physical and spatial intermediate representations, although the inspected excerpt gives no downstream comparison metrics.