Source note
Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents
Vision Language ActionRobot Foundation ModelGeneralist Robot PolicyDexterous ManipulationRobot Data Scaling
Summary
Harness VLA turns a frozen vision-language-action model into a reusable local manipulation primitive and uses a memory-guided planner to handle grounding, staging, transport, retries, and long-horizon composition. It improves robustness under instruction, layout, household, and clean-to-randomized perturbations without VLA fine-tuning or deployment-time expansion of the primitive library.
Problem
- End-to-end VLAs often fail when object targets are re-bound, layouts change, tasks become longer, or contact attempts become unstable outside the training distribution.
- Analytic robot primitives handle transport and positioning well but struggle with irregular grasps, constrained placement, articulated objects, and other contact-rich actions.
- Reliable language-conditioned manipulation matters because deployment scenes and instructions vary, while a single failed contact can derail an entire task.
Approach
- A planner selects calls from a fixed JSON primitive library instead of emitting low-level actions directly.
- Analytic primitives handle target grounding, free-space motion, wrist and gripper control, navigation, release, and re-staging.
- The frozen VLA is exposed through
vla_act, which performs short, retryable bursts for local contact-rich actions such as grasping, insertion, fixture actuation, and constrained placement. - Task-specific memory stores successful primitive traces with symbolic spatial queries; global memory stores reusable success rules and failure models, including empty-grasp and false-success cases.
- The planner bootstraps on one reference task instance, then re-grounds the stored trace on held-out layouts and seeds without adding new primitives or fine-tuning the VLA.
Results
- On standard LIBERO, Harness VLA with Claude Code reaches 96.0% overall success, or 384/400 trials, compared with 95.3% for the frozen RLinf VLA baseline.
- On LIBERO-Pro perturbations, Harness VLA reaches 82.4% overall with Claude Code and 72.1% with Codex. The Claude Code result is 38.6 percentage points above the strongest reported prior baseline, RATS at 43.8%, and 32.4 points above the direct RLinf baseline at 50.0%.
- On RoboCasa365, Harness VLA with Codex reaches 55.4% overall versus 30.0% for RLDX-1, a 25.4-point improvement. It reaches 91.6% on Atomic-Seen and 56.3% on Composite-Seen.
- On RoboTwin clean-to-randomized transfer, the paper reports 58.4% success.
- The reported gains support the claim that planner-controlled staging and retries extend a frozen VLA beyond its original trajectory distribution, although the excerpt does not provide a complete ablation of every memory and primitive component.