Action-state alignment for robot learning

Bimanual manipulation teams can now test a specific data pipeline for cross-robot transfer: generate simulator rollouts, turn them into photorealistic videos, and keep the paired action labels for policy training. CRAFT gives this approach a concrete shape. It starts from a small real dataset, builds a digital twin, replays trajectories in simulation, and uses Canny-guided video diffusion so the generated videos preserve motion structure that still matches the actions.

The practical use case is a lab that has demonstrations on one dual-arm setup and wants a policy to work on another without collecting a fresh target-robot dataset. The reported transfer numbers are large enough to justify that workflow as an engineering project, not just a data augmentation experiment. In simulation, transfer from bimanual UR5 to bimanual Franka reached 82.6% on Lift Pot, 89.3% on Place Cans, and 86.0% on Stack Bowls with no target-robot demos. In real tests from xArm7 to Franka, it reached 17/20, 15/20, and 16/20 successes on three tasks. Those results beat both simple shadow baselines and a target-data baseline reported with 100 collected target demos.

A concrete next build is an internal augmentation job for one narrow task family such as pick-place or container handling, with the simulator trajectory, rendered control video, generated photorealistic video, and original action sequence stored together as one training record. The cheap check is whether a policy trained on source-robot data plus generated target-style demonstrations cuts the amount of target teleoperation needed for one new embodiment or camera setup.

Shared-workspace robot teams can add a behavior-switch detector as a control-time safety layer in front of a frozen VLA policy. UA-ToM is a concrete example of that design. It adds a 992K belief module to a frozen 7B backbone, tracks whether a collaborator has changed behavior mid-task, and stays within a 50 ms control budget with 7.4 ms of added inference time.

The operational problem is plain: a robot that continues on an old assumption after a human or partner robot changes strategy keeps moving into the wrong space. This paper ties the detector to a timing target that matters in deployment. At a tighter ±3-step window, about 150 ms in a 50 ms loop, UA-ToM reached 85.7% hard detection across 1,200 episodes. The safety effect is also direct: enabling switch detection cut post-switch collisions from 2.34 to 1.11 per episode, a 52% reduction.

The immediate workflow change is to evaluate collaborative manipulation controllers with a narrow switch window and post-switch collision counts, not only average detection scores under loose tolerances. A concrete pilot would put a small belief tracker beside an existing shared manipulation policy, trigger a conservative fallback or replanning mode on high switch probability, and log collision and close-range time after the switch. Teams working on handover, co-carry, or shared assembly can test this without retraining the base controller.

Medical robotics groups now have a concrete way to build closed-loop colonoscopy datasets that are useful for autonomy work, not just retrospective video analysis. OpenRC combines a low-cost robotic retrofit with synchronized logging of video, operator commands, actuation state, and 6-DoF distal tip pose. The data is aligned on a shared stack and released in LeRobot 2.1 format.

That changes the first useful build for labs working on robotic colonoscopy. A dataset can now include navigation, failure, and recovery episodes with action and state alignment tight enough for policy learning and control-aware evaluation. OpenRC reports 1,894 episodes over about 19 hours across 10 task variations, including 142 failure episodes and 141 recovery episodes. After alignment, residual lag between operator action and actuation state is 55.6 ms, and actuation-to-tip-pose lag is centered at 0.0 ms. The hardware cost is under $5,000 excluding the EM tracker, which lowers the barrier for replication.

A practical next step is a failure-recovery benchmark built from lumen loss, wall contact, and fold engagement episodes, with one baseline policy that predicts the next control command and one recovery policy that is only trained on rescue segments. The useful near-term audience is research teams that already have colonoscopy video but lack synchronized action and state logs for closed-loop experiments.