Counterfactual and trace-based tests for robot world models

Robot-learning teams can use successful real-to-sim conversions to create paired transitions from the same initial state: replay the recorded action, replace it with a null action, and perturb its timing or direction. Agentic Real2Sim already reconstructs geometry, object state, physical parameters, and trajectories as runnable episodes; DWM shows why the resulting counterfactuals matter, reporting a mean 13.1-point planning-success improvement when training separates persistent world effects from action-driven change on simulated benchmarks. This combination offers a practical route to test that decomposition on realistic interaction records without collecting matched counterfactuals on physical robots.

The cheapest check is to rerun the successfully reconstructed DROID episodes under null and perturbed actions, train with and without the paired supervision, and evaluate on held-out recordings containing sliding, rebound, or post-contact motion. Because only 48 of 100 episodes replayed successfully with Agentic Real2Sim’s best tested backend, results should also be stratified by reconstruction quality; otherwise simulator error could be mistaken for a learned world effect.

Real-to-sim engineers should add object masks, gripper traces, contact points, and placement targets to each episode’s replay record, then repair the first structured mismatch instead of relying mainly on an end-to-end replay score. Agentic Real2Sim exposes failures from perception, alignment, scene assembly, and physics through one simulator-in-the-loop workflow. RoboInter1.5 shows that these intermediate signals can be annotated at scale, while Masked Visual Actions demonstrates that pixel-space entity trajectories can condition scene-response prediction across embodiments. Together, they suggest a diagnostic that distinguishes an incorrectly aligned robot from a wrong contact event or object trajectory before another full replay is attempted.

A low-cost evaluation can reuse Agentic Real2Sim’s DROID-100 set: measure per-frame robot-mask, object-track, contact-time, and final-placement errors, then test whether those residuals predict the existing replay judgments and help an automated repair step choose the parameter family to revise. The useful outcome is not merely a higher visual score, but fewer failed repair iterations and better physical replay.

Benchmark maintainers can test visual action interfaces as a closed loop: infer robot motion from a desired object trajectory, feed that inferred motion back into the forward model, and score whether the predicted object reaches the requested state with the correct contact and placement. Masked Visual Actions uses one checkpoint in both directions, but image fidelity alone does not show that the inferred action and predicted consequence agree. RoboInter1.5 supplies the object, gripper, contact, affordance, and placement representations needed to score that agreement across varied manipulation episodes and robot arms.

The first test can mask robot motion in held-out clips, infer it from the observed object path, run the inferred path forward, and compare endpoint, contact timing, placement validity, and conventional video metrics with the recorded outcome. Reporting cycle errors by robot embodiment would reveal whether a visually shared interface is genuinely embodiment-flexible or only produces plausible-looking motion.