Recoleta
GitHub
Source note

CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models

Authors
Khoa Vo, Sieu Tran, Taisei Hanyu, Yuki Ikebe, Duy Nguyen, Bui Duy Quoc Nghi, et al.
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionLong Horizon ManipulationNon Markovian PlanningSemantic Graph StateCode As Planner
Open arXivMarkdown source

Summary

CodeGraphVLP targets long-horizon robot manipulation where the next action depends on earlier observations, not just the current camera frame. It combines a persistent semantic graph, a one-time synthesized code planner, and object-focused prompting for a VLA policy.

Problem

  • Standard Vision-Language-Action models usually act from the latest observation, which fails on non-Markovian tasks where key evidence may be occluded or only visible earlier.
  • History-based extensions can miss sparse past evidence or add latency and compute cost as the context window grows.
  • VLM-in-the-loop hierarchical planners can improve long-horizon reasoning, but repeated model calls are slow and language-only subtask prompts still leave visual grounding brittle in clutter.

Approach

  • The system builds and updates a persistent semantic graph of task-relevant objects, attributes, and relations across time using segmentation, relevance filtering, cross-view association, tracking, and rule-based relation induction.
  • An LLM is called once at task start to write a task-specific Python planner that reads the graph, checks progress with simple predicates, stores lightweight task memory, and outputs the next subtask plus the relevant objects.
  • The executor VLA does not see the full cluttered scene. It receives a short subtask instruction and images masked to keep only the planner-selected objects.
  • Training matches deployment: recorded demonstrations are converted into subtask-conditioned, masked observations, and the VLA is fine-tuned by imitation learning on those inputs.

Results

  • On three real-world tabletop tasks, CodeGraphVLP reports an average success rate of 81.7%, above Gr00T N1.5 + Multi-frame: 56.7%, Gr00T N1.5: 31.7%, π0: 30.0%, π0.5: 5.0%, and π0 FAST: 0.0%.
  • On Pick-and-Place Twice, CodeGraphVLP reaches 80% full success and 100% on the intermediate "PnP Once" metric, versus 75% / 100% for Gr00T N1.5 + Multi-frame and 35% / 50% for Gr00T N1.5.
  • On Place-and-Stack, CodeGraphVLP reaches 80% success and 95% on the intermediate "Drop Cube" metric, versus 50% / 50% for Gr00T N1.5 + Multi-frame and 40% / 40% for Gr00T N1.5.
  • On Swap Cups, CodeGraphVLP reaches 85% success and 100% on the intermediate "Stage Cup" metric, versus 45% / 90% for Gr00T N1.5 + Multi-frame and 20% / 70% for Gr00T N1.5.
  • The paper also claims substantially lower planning latency than VLM-in-the-loop planning, but the excerpt does not include the actual latency numbers.
  • Training data size in the real-world setup is 100 demos for Pick-and-Place Twice, 100 for Place-and-Stack, and 200 for Swap Cups.

Link