Operational Robot Learning

A robot team running more than one manipulator should treat deployment logs as training input, including failed autonomous attempts, partial progress, recoveries, and human interventions. LWD gives a concrete pattern: send current checkpoints to a shared fleet, collect rollouts into an online replay buffer, mix them with offline data, and retrain a shared VLA policy with value learning that can propagate sparse rewards across long tasks.

The cheap adoption test is a narrow one: pick two or three tasks where the pretrained policy fails after small object, placement, or user-instruction changes, then run a short replay loop with success, failure, and intervention labels. The LWD result is still a research result, but its evaluation used 16 dual-arm robots across 8 real-world tasks, including 3–5 minute manipulation tasks, and reported 95% average success after a few hours of online interaction. That is enough evidence to justify building the data plumbing before buying another round of clean demonstrations.

Long-horizon manipulation policies need a visible task plan that operators and evaluators can inspect before execution. IVLR shows a buildable version: generate a full-task trace once from the initial image and instruction, pair each subgoal with an RGB keyframe, cache the trace, and condition the action decoder on the trace plus live camera observations.

This is most useful for tasks where step order and target placement are common failure points, such as moving several objects, using a tool before a final placement, or preparing a sequence of items. The first test should log the generated trace, the execution video, and the failure point, then compare no-trace, text-only, vision-only, and full trace variants. On LIBERO-Long, IVLR reports 37.7% success without traces, 62.0% with text-only traces, 68.4% with vision-only traces, and 92.4% with full interleaved traces. The current limit is practical: full trace generation takes about 10 seconds on one NVIDIA H20 GPU, so this fits tasks with planning slack before motion begins.

Teams using ACT-style imitation policies can add explicit 2D attention-point tracking to reduce visual drift without moving to a slower controller. MSACT keeps ACT’s action chunking and low-latency execution, then extracts task-relevant points from camera images and trains the policy to predict future attention-point sequences. The key engineering detail is that the attention supervision is self-supervised from future frames, so it does not require manual keypoint labels.

This retrofit is a good fit for bimanual fine manipulation and mobile manipulation where missed contacts, camera viewpoint changes, and distractors break dense visual features. In ALOHA bimanual trials, MSACT reports 53.00% success across 400 real-world trials, compared with 23.25% for ACT, while latency stays almost unchanged at about 45 ms. A related stereo mobile-manipulation system reports 85.0% average success across four real-world tasks, compared with 46.0% for ACT, and also performs better under disturbance tests. A practical validation run can keep the existing ACT baseline and add attention drift, inference latency, and randomized visual disturbance checks to the task scoreboard.