Recoleta
GitHub
Source note

ST- π π : Structured SpatioTemporal VLA for Robotic Manipulation

Authors
Chuanhao Ma, Hanyu Zhou, Shihan Peng, Yan Li, Tao Gu, Luxin Yan
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionRobotic ManipulationSpatiotemporal ReasoningLong Horizon ControlHierarchical Policy
Open arXivMarkdown source

Summary

ST-π\pi is a vision-language-action model for robotic manipulation that adds explicit spatiotemporal structure to both task planning and action generation. It targets long-horizon, multi-stage manipulation where standard VLA models often drift or lose track of sub-task boundaries.

Problem

  • Existing VLA and 4D-VLA systems usually predict actions from fused observations in an implicit way, which makes long sequences of sub-tasks hard to separate and execute reliably.
  • Fine-grained manipulation needs two things at once: explicit sub-task boundaries at the planning level and stable step-by-step control inside each sub-task.
  • This matters for assembly and household tasks where errors accumulate across stages and later actions depend on earlier ones being completed in the right place and order.

Approach

  • The model has two parts: ST-VLM for chunk-level planning and ST-AE for step-level control.
  • ST-VLM builds a 4D representation from image sequences, geometry features, and timestamps, then predicts a sequence of chunk-level action prompts. Each prompt includes semantic intent, a spatial target, and a temporal duration for one sub-task.
  • The planning module uses causal attention across prompts so later sub-tasks depend on earlier ones, which gives an explicit ordered decomposition of a long task.
  • ST-AE takes one chunk-level prompt at a time and generates low-level action steps with two generators: a spatial generator for trajectory shape and a temporal generator for step order and consistency.
  • Action generation uses flow matching, with a time-dependent fusion that shifts guidance from spatial shaping early in denoising to temporal refinement later.

Results

  • The paper introduces STAR, a real-world dataset on a Franka Research 3 robot with 30 manipulation tasks, 50 demonstrations per task, and about 300k interaction steps. It includes sub-task descriptions, target locations, and execution durations.
  • On the shown benchmark table, 4D-VLA is the strongest listed baseline with 88.6% average success rate. ST-π\pi claims consistent gains over prior VLA baselines, but the provided excerpt does not include the full ST-π\pi result row, so its exact average improvement is not visible here.
  • Visible baseline numbers show why long-horizon settings are hard: OpenVLA gets 53.7% success on the Long suite, Octo 51.1%, SpatialVLA 55.5%, TraceVLA 54.1%, while 4D-VLA reaches 79.1%.
  • The excerpt also reports completion time for several baselines. For average completion time, OpenVLA has 8.0s, Octo 7.0s, and SpatialVLA 6.6s. The ST-π\pi completion-time numbers are not visible in the provided text.
  • Strong concrete claim: the method improves long-horizon manipulation by combining explicit chunk-level sub-task planning with dual-generator step-level action refinement, rather than relying on implicit spatiotemporal representations alone.

Link