Coordinate-aligned supervision for robot policy training

VLA teams pooling demonstrations across camera setups need policies that predict physical consequences without relearning camera-to-robot transforms. WALA gained 21 points on RoboCasa by supervising semantic and geometric future deltas; robot-centric pointmaps improved π₀.₅ by 7.6 points and widened their advantage at unseen camera placements. Together, these results suggest expressing future geometry in the robot frame before latent-action training.

Add a decoder that predicts future robot-centric pointmaps or their deltas, centered on the current end effector. Keep WALA’s semantic target, and train the action head against the same coordinate system. This directly couples predicted scene evolution to executable motion.

Run a low-data test with fixed-camera training and unseen-camera evaluation. Compare depth-delta, camera-frame pointmap-delta, and robot-frame pointmap-delta targets. Use a pilot decision threshold: stop if the robot-frame target improves unseen-camera success by under 5 percentage points or lowers fixed-camera success by more than 3 points.

Teams generating robot training videos need a cheap way to reject visually plausible trajectories that violate camera calibration or gripper kinematics. Xiaomi-Robotics-U0 reports a real-policy OOD gain from 36.9% to 63.2% with generated data, while Pix2Act shows that continuous gripper-keypoint paths in two image planes can be triangulated into precise 3D actions.

Attach projected gripper keypoints to each generated view, triangulate them frame by frame, and reject clips whose recovered pose disagrees across views, exceeds joint limits, or breaks temporal continuity. The same paths can condition generation, giving the model an explicit geometric control signal and yielding an audit score for every synthetic clip.

Generate matched batches with and without this check, then train equal-size policies and evaluate unseen scenes and camera poses. Use a pilot decision threshold: drop the check if it rejects fewer than 10% of clips while improving OOD success by under 3 percentage points, or if accepted clips still exceed the calibration error of real demonstrations by more than 25%.

Dexterous robotics teams adapting a tool skill to a new hand need reward signals beyond one retargeted trajectory. Regrind preserves hand-object relationships from one human demonstration and refines the motion with residual RL. WALA shows that action-free videos can supply semantic and depth-based future-change supervision, including a large gain when labeled robot data are scarce.

Pretrain a future-delta encoder on ordinary videos of the same tool interaction. During Regrind simulation, score rollouts by agreement with the predicted object-motion and contact-region changes, alongside its object-centric keypoint tracking reward. This could widen the useful restart distribution without requiring motor labels for every human example.

Test one scissors or screwdriver skill with identical retargeting and simulation budgets. Measure success under perturbed object pose, tracking error, and hardware completion. Use a pilot decision threshold: stop if video-derived rewards add under 5 percentage points of perturbed-pose success, increase object-tracking error by over 2 mm, or reduce hardware completion relative to keypoint rewards alone.