Contact-Aware Robot Policy Control

Robot policy teams working on fine manipulation should add a geometry path to the action decoder and test it on tasks where semantic object recognition is already good but control fails: transparent bottles, ring-shaped objects, tight insertions, stable release, and contact-heavy alignment.

GeoAlign gives a concrete implementation pattern. It post-trains a Depth Anything V2-Small branch on robot RGB-D data, discards the depth head at rollout, and uses RGB-derived geometry features. The robot state creates query slots that select local geometry from the feature grid, then compact geometry tokens condition the flow-matching action decoder. This keeps rollout inputs to RGB, language, and proprioception while still giving the action head local shape cues.

The cheap validation is a task slice with geometry-sensitive objects. GeoAlign reports 78.8% average success on eight real ALOHA tasks, compared with 65.0% for a matched RGB-only baseline, with transparent-bottle success rising from 35.0% to 75.0% and tape-roll insertion from 40.0% to 65.0%. GeoSem-WAM points in the same direction for world action models: future geometry and semantic supervision are used during training, dense heads are removed at deployment, and real Franka success rises from 88.9% for Fast-WAM to 95.4%.

A deployment workflow can add a short adaptation step before a frozen VLA policy is used in a new workspace. The robot collects interaction data in the target setting, keeps the policy backbone fixed, and updates only learned latent prompt tokens using a self-supervised state-grounding loss that predicts end-effector position and gripper state.

TTT-VLA is the concrete case to copy first. It adds a latent prompt to the policy conditioning input during training, then adapts that prompt at test time with data from the current environment. On SimplerEnv WidowX single-embodiment tasks, π0.5 improves from 51.1% mean success to 63.5% with state-grounded latent prompts and to 67.4% after test-time prompt optimization. The method also improves Google Robot visual matching from 67.5% to 72.4% after test-time training.

This is most useful for teams that already trust a base policy but see failures after camera, lighting, object layout, or embodiment changes. The reported compute is high, 15–30 minutes on 8 NVIDIA H100 GPUs for the listed experiments, so the practical first check is an offline replay or simulator trial that measures whether prompt-only updates correct a small number of repeated decision errors without changing the policy weights.

Teams fine-tuning VLA policies on a sequence of manipulation skills should store replay data by sub-skill phase, not only by trajectory or task. The operational pain is clear: brief contact phases such as grasping can be under-sampled when replay frames are drawn uniformly, even though those frames decide whether the full task succeeds.

PHASER splits trajectories into phases such as approach, grasp, and transport, gives each phase an equal frame budget, and routes replay toward old phases likely to conflict with the current phase. The routing uses language similarity, visual similarity, and action divergence, with scores computed once at each task transition. Auto-PC can propose phase boundaries using action-signal change-point detection plus VLM semantic verification, reducing the need for manual labels.

The first implementation can be a buffer change around an existing fine-tuning pipeline: segment demonstrations into phases, reserve equal memory per phase, and compare against uniform experience replay on a sequential LIBERO-style suite. PHASER reports 85.8% average success on LIBERO-Long with OpenVLA-OFT-7B, compared with 54.6% for standard experience replay. The same paper reports large gains across QwenGR00T-3B and QwenOFT-3B, which makes the buffer design worth testing before changing the policy architecture.