Source note
RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
Summary
RoboLab is a high-fidelity simulation benchmark for testing whether real-world-trained robot policies actually generalize to new tasks and domains. It builds held-out simulated tasks, scores failure modes and motion quality, and shows that current vision-language-action policies perform poorly on this kind of out-of-domain evaluation.
Problem
- Existing robot benchmarks often train and test in overlapping simulated domains, so success rates can saturate and hide weak generalization.
- Real-world evaluation is expensive and slow, while many simulators are either too low-fidelity for real-world policies or too costly to extend scene by scene.
- Binary success alone misses useful failure signals such as wrong-object grasps, drops, collisions, and sensitivity to camera or scene changes.
Approach
- The paper introduces RoboLab, a simulation platform in IsaacLab/Isaac Sim that generates photorealistic scenes and language-conditioned tasks with human-written or LLM-assisted workflows.
- It defines RoboLab-120, a benchmark of 120 human-verified tasks across three competency axes: visual, procedural, and relational, each with simple, moderate, and complex levels.
- Policies are trained only on real-world data such as DROID and then evaluated in held-out simulation, which separates training from test domains.
- Evaluation goes beyond success rate with graded task scores, logged failure events, trajectory metrics such as SPARC, speed, and path length, plus sensitivity analysis using simulation-based inference to estimate which scene variables drive success or failure.
- The generation pipeline scales scene and task creation: the paper reports over 800 generated scenarios and evaluates 812 generated tasks across 59 scenes for instruction-condition alignment and feasibility.
Results
- On RoboLab-120, overall performance is low. Table I reports overall success of 23.3% for π0.5, 15.7% for π0-FAST, 5.2% for π0, 2.0% for GR00T N1.6, and 1.5% for PaliGemma.
- The paper also states the best policy reaches 31.9% success overall, while the introduction gives an approximate figure of ~30% for π0.5. The excerpt contains this inconsistency, but both numbers support the same claim: strong current models fail on most held-out tasks.
- For π0.5, success by difficulty is 26.3% / 23.2% / 11.7% on simple / moderate / complex tasks.
- For π0.5, relational subtasks are much easier than most other axes: 56.2% overall relational success in Table I, with text-reported breakdowns of 76.0% on conjunction, 60.0% on counting, and 23.9% on spatial relations.
- For π0.5, visual grounding remains weak: the text reports 35.0% for size, 30.0% for color, and 21.5% for semantics. Procedural skills are also weak: 53.3% on reorientation, 20.0% on affordances, and 16.0% on stacking.
- Language robustness is brittle. Table IV shows overall success across language specificity levels for π0.5 at 16.8%, 23.3%, and 25.8%; for π0-FAST at 9.7%, 15.7%, and 15.2%; and for π0 at 3.4%, 5.2%, and 6.5%.
- Scene complexity hurts performance. In one ablation, π0.5 drops from 70% success with 1 target object to 30% with 2 and 20% with 3 on a canned-food packing task.
- Instruction wording can flip outcomes in the same scene. Examples for π0.5 include 80% on "Put the white mugs in the grey bin" versus 0% on "Put away mugs," and 50% on "Take all the bananas out of the grey bin and put it on the table" versus 10% on "Empty the grey bin."
- The task-generation pipeline also has quantitative validation: across 812 generated tasks in 59 scenes, the paper reports 0.91 instruction-condition alignment, 0.96 clarity, 0.92 feasibility, 0.95 semantic match, 76% fully aligned tasks, about 1% misaligned, and 88% object coverage.
- On selected real-vs-sim comparisons over 6 simple tasks, Table V reports real success rates of 79.5, 34.1, 63.2, and 0.0 versus simulated rates of 74.0, 42.0, 18.0, and 4.0 for the shown policies, supporting the claim that simulation can be a useful but imperfect proxy for real-world behavior.