Deployment reliability benchmarks

Robot teams deploying VLA policies on real hardware can now justify a runtime pass that changes execution scheduling before they touch model size. StreamingVLA shows that overlapping observation, action generation, and execution cuts time per action on LIBERO from 74.5 ms to 33.7 ms while holding average success at 97.1%. Its adaptive early observation path reduces the halting gap from 232.3 ms to 36.0 ms, with success at 94.9%, which is a usable tradeoff for robots whose failures come from stop-and-go motion more than raw policy weakness. The build is concrete: expose halting gap and time-per-action as first-class deployment metrics, then add an asynchronous inference runner that can execute single-step actions as soon as they are generated and only refresh observations early when predicted saliency is low. A cheap check is to replay your current controller on one contact-sensitive task and measure whether pauses between action bursts, not wrong actions, are the dominant source of misses. If they are, a streaming executor is a more direct fix than retraining a larger policy.

Robot evaluation pipelines need a paraphrase test set before claiming instruction following is stable. LIBERO-Para changes only the wording of commands and still finds 22.8 to 51.9 point success drops across seven VLA configurations. PRIDE also lands 8.4% to 22.0% below raw success, which means binary completion scores miss a large share of language brittleness. The practical workflow change is simple: keep a fixed paraphrase bank for each production task, split it across action wording and object reference changes, and report both task success and a robustness score that weights harder rewrites more heavily. The object side deserves special attention because common name substitutions alone produce 19.8 to 51.0 point drops. Teams fine-tuning on narrow in-house demonstrations can use this as an acceptance gate before field trials, especially for warehouse and home tasks where operators rarely repeat the training phrasing exactly.

Shared real-world evaluation rigs are becoming practical enough to use as an external test service for generalist manipulation models. ManipArena fixes the embodiment, collects 20 physical tasks with 10,812 expert trajectories, and requires each participant to submit one model endpoint for all tasks. Its trial structure separates in-domain runs, shifted but in-distribution runs, and semantic OOD runs under controlled lighting and booth conditions. That setup fits a concrete adoption path for labs and startups that already benchmark in simulation but need one comparable physical test before customer pilots or paper claims. The useful build here is not another simulator. It is a remote evaluation workflow with one frozen endpoint, matched task assets, and standard logs that include low-level motor signals alongside camera streams. The immediate value is comparable failure reports across models and labs, especially for systems that look good on LIBERO but have not been tested against real latency, contact noise, or longer mobile-manipulation episodes.