Physical Rollout Evaluation for Robot Policies

Robot labs collecting bimanual dexterous demonstrations should add a clip-level quality score before training policies. Dexora shows a concrete version: two 6-DoF arms, two 12-DoF hands, four RGB views, joint states at 20 Hz, a matched MuJoCo twin, and a diffusion policy whose loss is weighted by an offline discriminator that down-weights poor demonstrations.

The pain is operational. High-DoF teleoperation data includes operator variation, tracking errors, occlusion, and latency. Throwing all episodes into training treats unstable clips as useful supervision. A small discriminator or evaluator that scores completion, contact stability, object pose drift, and hand-tracking confidence can turn an ordinary data lake into a trainable corpus.

A cheap first test is to rescore an existing teleoperation set, train the same policy with and without quality weights, and compare physical rollouts on tasks that need finger articulation, such as cap twisting, pen use, book retrieval, or dough handling. Dexora’s reported gap on dexterous tasks, 66.7% average success versus 51.7% for GR00T N1 and 6.7% for Diffusion Policy, gives this workflow enough signal to try on smaller local rigs.

VLA deployment teams should add a camera-corruption regression suite before physical trials. StableVLA documents the failure mode clearly: policies that work under clean camera inputs can break under blur, noise, weather-like effects, and digital artifacts that were absent during training.

The practical build is small. Take the team’s normal validation clips or simulation rollouts, apply fixed severity levels for common visual corruptions, and report success by task family. For policies using a VLA-Adapter-style bridge between the vision encoder and the language-policy stack, test a replacement visual projector based on StableVLA’s Information Bottleneck Adapter. Its design filters noisy feature channels while preserving spatial detail through a parallel path, with fewer than 10M added parameters and no extra robot data.

The acceptance test should include the policy’s normal clean score, its worst corruption cases, and at least one real robot trial with induced camera degradation. StableVLA reports LIBERO severity-5 gains such as Object at 70.2% versus 29.3% and Long at 45.3% versus 26.2%, plus a real Pack Doll result of 50% success versus 20% for VLA-Adapter-0.5B.

Teams evaluating dexterous tabletop robots should score whether the table remains usable after each action. DexHoldem gives a direct template: physical rollouts use a four-level rubric that separates scene-preserving success, disruptive completion, task failure, and disruptive failure. The distinction matters because a hand can complete a card or chip action while leaving the next decision state damaged.

This is easy to add to compact tabletop domains. Define legal end states, object displacement limits, chip or part inventory checks, and recovery triggers. Record task completion and scene preservation separately. For systems with tactile sensors or learned dynamics, add a contact-utility check that compares tactile reconstruction with downstream action success.

WorldArena 2.0 shows why this extra score is needed for contact-rich models. Wan2.2 has the best tactile prediction quality on UniVTAC, with 21.26 PSNR and 0.746 SSIM, yet averages 50% task success across Insert HDMI and Lift Bottle. A useful evaluation harness should pair tactile or video quality with physical rollout success, because clean reconstruction can still fail as control evidence.