Robot policy deployment benchmarks

Robotics teams trying to run Pi 0.5 or GR00T-class policies close to the robot should add a 4-bit policy acceptance test before buying more inference hardware. The test is concrete: quantize the language backbone and diffusion action head to uniform W4A4, calibrate DiT activation scales on a small set of unlabeled trajectories, then compare against the FP16 policy on the team’s manipulation suite.

Ω-QVLA reports Pi 0.5 W4A4 at 98.0% average LIBERO success versus 97.1% FP16 and GR00T N1.5 W4A4 at 87.8% versus 87.0%, with a 71.3% static memory-footprint cut. The practical gate should include task success, action smoothness, memory use, and real robot progress, since the same paper reports ARX R5 dual-arm progress close to FP16 and a much larger gap over QuantVLA.

Manipulation teams handling fragile, slippery, or deformable objects should add contact-force metrics to rollout evaluation. A useful scorecard records maximum transient grip force, average grip force, maximum transient applied force, and average applied force for each task attempt, with language-conditioned runs such as “gently” and “firmly.”

Tabero shows how to build the data path by replaying manipulation trajectories in Isaac Lab with tactile sensing, recording GelSight-style tactile images, marker displacement grids, and fingertip forces, then sending predicted pose and force targets to a hybrid controller. It reports over 70% lower average grip force under gentle instructions. CoP adds a complementary path for dexterous hands: represent touch as a 3D contact force and 3D contact location per tactile region, then train in simulation. On real peg-in-hole insertion, CoP reaches 0.78 success across six shapes, compared with 0.48 for raw taxels and 0.53 for binary contact. A rollout can pass task completion and still fail the contact-force limit.

Robot data teams with task-level demonstrations can add an offline primitive-labeling pass before VLA fine-tuning. The workflow is specific: infer a sequence over primitives such as Grasp, Place, Lift, Move, Push, Pull, Insert, Press, Twist, Tilt, and Rotate; generate state-based boundary rules for those segments; track object masks with SAM and Cutie; train the VLA on canonical primitive instructions; then use a planner and state-history switch rules during execution.

PrimitiveVLA reports large gains where memorizing whole demonstrations hurts most. OpenVLA rises from 7.38% to 45.50% on LIBERO-90-Novel, OpenVLA-OFT reaches 66.50% on LIBERO-Long versus 3.75%, and pi0.5 reaches 80.25% on LIBERO-Long versus 30.50%. The cheap check is a held-out set of novel object-skill pairs and long-horizon tasks with the same demonstration budget.