--- kind: ideas granularity: day period_start: '2026-04-18T00:00:00' period_end: '2026-04-19T00:00:00' run_id: 006dd63e-d43e-4c5c-97bc-f01a25413d82 status: succeeded topics: - robotics - long-horizon manipulation - benchmarks - vision-language-action - robot learning tags: - recoleta/ideas - topic/robotics - topic/long-horizon-manipulation - topic/benchmarks - topic/vision-language-action - topic/robot-learning language_code: en pass_output_id: 79 pass_kind: trend_ideas upstream_pass_output_id: 78 upstream_pass_kind: trend_synthesis --- # Robot rollout diagnostics ## Summary Real-world long-horizon robot evaluation is getting specific enough to change day-to-day workflow. The clearest near-term moves are stage-wise internal evals, structured failure review after rollouts, and a dynamic-grasp stress test in release gates. The common point across the evidence is operational: larger robot data and deployment loops need better ways to see where execution breaks over time, not just whether a run ended in success. ## Stage-wise long-horizon robot evaluation for internal model testing Robotics teams now have enough evidence to replace single success rates with stage-wise long-horizon evaluation in weekly model tests. LongBench shows why this matters: long tasks fail through different mechanisms, including execution drift, phase-transition errors, missed timing windows, and ambiguous states that require memory of earlier context. A single pass/fail number hides those differences. The practical build is a small internal eval harness that scores each task by completed sub-steps and tags failures by mechanism. Teams shipping tabletop manipulation policies can start with a narrow slice: one temporal-window task, one error-accumulation task, and one context-dependent task. The point is to see whether a model stalls at the same stage every run, degrades late in the rollout, or confuses visually similar states. That gives model and data teams a clearer target than aggregate success. LongBench also gives a concrete warning about where current policies break. On fully observable tasks, pi_0 averages 86.3 stage-wise score, while Diffusion Policy reaches 51.2 and OpenVLA-OFT 32.7. Dynamic grasping is the sharpest stress case: pi_0 reaches 73.3, while the other listed systems stay between 0.0 and 13.3. A cheap validation step is to rerun one existing in-house task with stage annotations and compare the ranking against the current pass/fail leaderboard. If the ordering or failure story changes, the old eval is hiding useful signal. ### Sources - [LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks](../Inbox/2026-04-18--longbench-evaluating-robotic-manipulation-policies-on-real-world-long-horizon-tasks.md): LongBench summary states the benchmark separates execution and context difficulty, uses stage-wise scoring, and reports policy gaps on long-horizon tasks. - [LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks](../Inbox/2026-04-18--longbench-evaluating-robotic-manipulation-policies-on-real-world-long-horizon-tasks.md): The abstract describes mechanism-aware evaluation for execution robustness, temporal consistency, and context-dependent reasoning in real-world manipulation. - [LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks](../Inbox/2026-04-18--longbench-evaluating-robotic-manipulation-policies-on-real-world-long-horizon-tasks.md): The introduction explains that aggregate success fails to distinguish execution instability, dynamic response limits, and contextual ambiguity. ## Failure-mode labeling workflow for real robot rollout review A failure review tool for robot rollouts is now easier to justify than another round of aggregate benchmarking. LongBench organizes long-horizon manipulation by mechanism, including phase dependence, iterative progress, error accumulation, temporal windows, completion ambiguity, count ambiguity, subtask-branch ambiguity, and cross-episode ambiguity. That taxonomy is enough to structure a review workflow for real deployments. The useful product here is a post-run debugger that aligns video, action chunks, and task stages, then asks the reviewer to label the failure mode from a short fixed set. Teams working on VLA or VA policies could use it after every batch of real-world trials. Over a few weeks, they would get a distribution of failure causes by task and policy version, which is more actionable than a single success score and cheaper than full relabeling of every trajectory. This fits the current deployment cycle described in the broader robot learning coverage. The field is already training on larger real-world datasets and improving models with deployed robot feedback. Google spent 17 months collecting data across 700 tasks for RT-1 and reported 97% success on seen tasks and 76% on unseen instructions. Once teams are running collection loops at that scale, the bottleneck moves to deciding which failures deserve new data, a controller change, or a task redesign. A lightweight failure labeling tool gives that triage layer. ### Sources - [LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks](../Inbox/2026-04-18--longbench-evaluating-robotic-manipulation-policies-on-real-world-long-horizon-tasks.md): LongBench defines concrete mechanism categories for long-horizon failures and motivates evaluation beyond aggregate success. - [LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks](../Inbox/2026-04-18--longbench-evaluating-robotic-manipulation-policies-on-real-world-long-horizon-tasks.md): The paper abstract frames the benchmark around execution robustness, temporal consistency, and context-dependent reasoning. - [Robots learn: A brief, contemporary history](../Inbox/2026-04-18--robots-learn-a-brief-contemporary-history.md): The article reports RT-1 data collection over 17 months and 700 tasks, supporting the claim that larger deployment and data loops raise pressure for better failure triage. ## Dynamic-grasp acceptance test in robot release gates Dynamic grasping belongs in acceptance testing before a long-horizon manipulation policy goes anywhere near a live workflow. LongBench shows that this temporal-window task separates current systems much more sharply than easier fully observable tasks. pi_0 scores 73.3, while OpenVLA-OFT scores 0.0, SmolVLA 10.0, Diffusion Policy 13.3, MemoryVLA 10.0, and CronusVLA 13.3. That spread is large enough to use as a gate. For teams evaluating warehouse pick, tote transfer, or moving-object handoff policies, the workflow change is simple: add one short dynamic-grasp test to every release candidate and require a minimum stage-wise score before field trials. The test is cheap because it focuses on a narrow capability with clear operational consequences. If the policy misses timing windows on a controlled benchmark, it is likely to lose reliability when conveyors, human handoffs, or object motion add noise. The broader market context supports a stricter gate. Commercial deployment pressure is rising, with the Technology Review piece citing $6.1 billion in humanoid robot investment in 2025 and growing use of deployed systems for data collection and improvement. A release process that includes temporal-window stress tests is a straightforward way to catch a known weak point before operators absorb the failure cost. ### Sources - [LongBench: Evaluating Robotic Manipulation Policies on Real-World Long-Horizon Tasks](../Inbox/2026-04-18--longbench-evaluating-robotic-manipulation-policies-on-real-world-long-horizon-tasks.md): LongBench reports dynamic grasping as the hardest fully observable task and gives the stage-wise scores across six policies. - [Robots learn: A brief, contemporary history](../Inbox/2026-04-18--robots-learn-a-brief-contemporary-history.md): The article notes rising commercial deployment and investment, which increases the need for concrete pre-deployment acceptance tests.