Source note
Active Real-World Factor-Based Evaluation for Generalist Robot Policies
Robot Policy EvaluationGeneralist Robot PolicyActive LearningReal World RoboticsFactor Based Testing
Summary
The paper presents an active real-world evaluation framework that estimates how a generalist robot policy performs across task-factor variations while reducing the number of physical trials. Across 2,331 evaluations of three manipulation tasks, it reports typically saving 20–40% of trials compared with uniform-random testing.
Problem
- Generalist robot policies can fail under changes in object position, table height, camera viewpoint, and other deployment conditions, but exhaustive real-hardware evaluation across these factors is slow and costly.
- Narrow test suites and aggregate success rates can miss failure-prone regions and misrepresent a policy’s performance distribution and deployment readiness.
Approach
- Represent each evaluation configuration by structured task factors and model the policy’s continuous performance score over the resulting design space.
- Fit a probabilistic surrogate, primarily a Gaussian Process with an RBF kernel and automatic relevance determination, to predict performance and uncertainty from previously evaluated configurations.
- Use Bayesian active-testing acquisition functions, including posterior standard deviation, negative integrated posterior variance, BALD, and EPIG, to select the next most informative real-world configuration.
- Begin with 30 randomly selected evaluations, then sequentially evaluate selected configurations and update the surrogate until the trial budget is exhausted.
Results
- The study covers 2,331 valid real-world configurations across three UR5e manipulation tasks, three factors, an 11×11 object-position grid, three table heights, and three scene-camera viewpoints.
- It reports more than 700 ground-truth real-world evaluations per task; each evaluation took roughly one minute.
- With a 100-trial evaluation budget, active testing typically saves 20–40 trials, or 20–40% of the work, relative to typical random testing.
- The excerpt does not provide detailed numerical surrogate-prediction errors or per-method results; its strongest concrete claim is that active testing more efficiently characterizes the policy’s performance distribution and identifies failure-prone regions than random testing.