Recoleta
GitHub
Source note

Prompt-Driven Exploration

Authors
Sunshine Jiang, John Marangola, David Zhang, Raghuram Kowdeed, Ruiyang Luo, Nitish Dashora, et al.
Source
arXiv
Published
Collection
Embodied AI
Robot Foundation ModelVision Language ActionPrompt ExplorationSparse Reward RLRobot Data Scaling
Open arXivMarkdown source

Summary

Prompt-Driven Exploration (PDE) explores robot behavior by changing the language prompt instead of adding only action noise. A vision-language model rewrites prompts from rollout videos, helping reinforcement learning find successful behaviors when the initial policy has near-zero reward.

Problem

  • Sparse binary rewards leave weak vision-language-action policies with no successful trajectories to reinforce, especially on difficult manipulation tasks.
  • Action-space noise changes individual actions locally, so it rarely escapes a policy that follows the wrong strategy across an entire episode.
  • The problem matters for fine-tuning generalist robot policies from limited demonstrations, where initial success can be close to 0%.

Approach

  • PDE samples alternative prompts for the same task and keeps the prompt fixed during each rollout, creating globally different behavior modes.
  • A vision-language model watches rollout videos, diagnoses failures, and proposes revised prompts without updating model weights during prompt search.
  • The method treats the prompt sampler as an implicit posterior over useful policy behaviors and updates it with task outcomes and trajectory history.
  • Proximal Policy Optimization trains on exploratory and canonical prompts through mixture sampling and mixed backpropagation, transferring useful behavior back to the canonical task prompt.

Results

  • In a microwave-closing case study, prompt search found 10-12 successful prompt variants within 85 cumulative rollouts; the best prompts reached below 40% success before policy training.
  • PDE increased success on the original canonical prompt from 0% to approximately 98% by training step 100, while standard action-noise PPO stayed near zero.
  • On LIBERO-PRO, PDE outperformed all tested baselines across hard, medium, and easy tiers, with a 60% relative improvement on the hard tier of 47 tasks.
  • The benchmark covered 120 LIBERO-PRO tasks and evaluated success over 250 environments per task; PDE also improved learning over action noise with GR00T and Pi0 backbones.
  • The paper reports broader gains on ManiSkill manipulation tasks and challenging LLM coding tasks, supporting use beyond one robot benchmark.

Link