VLA evaluation gates

Robot teams evaluating a VLA policy should add a delay sweep to their normal task-success tests. The test should record end-to-end milliseconds per control step, VLM backbone call frequency, action chunk length, and success at fixed delays such as d=0, 2, 4, 8, 15, and 20. This catches a failure mode that single success-rate tables hide: the robot may keep moving while the next action chunk was computed from an old observation.

Two current papers give useful baselines for this check. Latent Bridge reports near-synchronous LIBERO success while cutting GR00T-N1.6-3B from 90 ms to 49 ms per step and pi_0.5 from 76 ms to 46 ms by predicting feature or KV-cache deltas between full VLM calls. The asynchronous-inference benchmark shows a separate control problem: at LIBERO delay d=20, A2C2 reaches about 58% success while a naive asynchronous baseline is around 10–12%. A practical adoption gate is to run the candidate policy through both measurements before any real-robot trial: latency with skipped backbone calls, then task success under delayed action chunks.

Teams with simulation trajectories can test a small, targeted video-transfer pass before collecting more real robot data. The concrete workflow is to select a coreset using action-prediction loss and visual diversity, generate realistic variants that preserve the same action trajectory, then train the VLA policy on the mixed data. The validation set should include changed object layouts, textures, lighting, and language instructions.

Seeing Realism from Simulation gives a useful scale for the first experiment. A 10% augmented coreset raised RoboTwin 2.0 Hard multi-task success for RDT-1B from 23.0% to 31.0%, and LIBERO-Plus pi_0 improved from 42.7% to 47.8%, with larger gains on object layout and instruction changes. The same paper reports small drops on standard LIBERO, so the check should target deployments where train and test visuals differ enough to matter.

Labs comparing closed or partially released robot policies can now build a local baseline around MolmoAct2. The minimum useful workflow is to run the released model and code on a small set of local tasks, then fine-tune or filter from the released SO-100/101, DROID, or BimanualYAM data sources using the same evaluation script. This gives teams a check on whether their robot, cameras, and task mix benefit from the open model before they spend time collecting a large local dataset.

MolmoAct2 is relevant because the release includes model weights, code, and training data, including 720 hours of bimanual YAM trajectories, 38,059 SO-100/101 episodes, and 74,604 filtered DROID successful episodes. The paper also names deployment blockers that local teams already face: costly hardware assumptions, latency from reasoning-heavy policies, and success rates below dependable use. The missing operational step is a reproducibility run on the lab’s own tasks with success rate, intervention count, and control latency logged together.