Stage-wise long-horizon evaluation with hazardous commit tracking
Benchmarks for long-horizon manipulation need a failure log, not just a success number. LongBench gives a concrete template: split tasks by failure mechanism, score progress stage by stage, and keep fully observable tasks separate from tasks that require memory of earlier context. The practical build is an evaluation harness for robot teams already running tabletop or lab manipulation. It should tag episodes with mechanism labels such as phase dependence, error accumulation, temporal windows, and ambiguity classes, then report where the policy stalled or drifted. That is more useful for model selection than a single task-completion rate, because LongBench shows large gaps on specific mechanisms even when all models use the same 16-step open-loop interface. The same scoring style also fits safety review. HazardArena shows why endpoint success hides risk: a policy can reach a hazardous precondition, or nearly complete an unsafe action, without finishing it. A deployment-facing test rig should therefore add action-level attempt, commit, and success counters for unsafe twins of common tasks. Teams shipping VLA systems into homes, labs, or warehouses can build this with their existing teleoperation and replay stack before adding new model complexity. A cheap first check is to re-score one current benchmark or internal task set with stage-wise and pre-hazard metrics and see whether the ranking of candidate policies changes.