Recoleta
GitHub
Source note

BiliVLA: Scene-Aware Vision-Language-Action Model with Reinforcement Learning for Autonomous Biliary Endoscopic Navigation

Authors
Jinsong Lin, Chi kit Ng, Zhiyong Xiong, Zikang Pan, Yihan Hu, Tabassum Tamima, et al.
Source
arXiv
Published
Collection
Embodied AI
Vision Language ActionSurgical RoboticsEndoscopic NavigationReinforcement LearningRobot Policy
Open arXivMarkdown source

Summary

BiliVLA is a vision-language-action policy for autonomous ERCP endoscope navigation in a phantom model. It joins target recognition, bounding-box grounding, and 3-DoF motor commands, then improves the policy with supervised tuning and GRPO.

Problem

  • ERCP cannulation needs stable navigation in a narrow monocular view with glare, occlusion, tissue contact, and variable anatomy.
  • Existing robotic or vision systems often split perception, planning, and control, which can fail when target appearance changes or the view degrades.
  • Safer autonomous navigation matters because repeated cannulation attempts raise clinical risk, including post-ERCP pancreatitis.

Approach

  • The model takes an endoscopic image plus a stage-specific instruction and outputs three items: target category, normalized bounding box, and a discrete action.
  • The robot action space has 11 motion primitives: 8 bending directions, forward, backward, and stop, mapped to 3 DoF actuation for tip bending and insertion or retraction.
  • The authors build BiliVLA-Motion with 10k image-motion pairs from a phantom ERCP setup: 3k entry navigation, 2.6k lumen traversal, and 4.4k calculus localization.
  • Scene-aware supervision ties each instruction to the target category and output schema, so the policy learns which structure to find in each ERCP stage.
  • Safety-aware labels set wall-contact frames to a full-image box and a backward command, teaching the policy to retreat when the view shows unsafe proximity.
  • Training has two stages: LoRA supervised fine-tuning of Qwen3-VL-8B with grounding and action labels, then GRPO using rewards for bounding-box IoU, correct action, and valid output format.

Results

  • In real-world phantom experiments across three ERCP subtasks, BiliVLA reports total mIoU 0.9625, action precision 91.96%, and success rate 84.85%.
  • Against EndoVLA, total action precision rises from 84.44% to 91.96%, and success rate rises from 58.86% to 84.85%.
  • Against Qwen3-VL, total success rate rises from 51.82% to 84.85%, with total mIoU rising from 0.8117 to 0.9625.
  • GRPO improves the full model over BiliVLA without GRPO: success rate 84.85% versus 63.64%, action precision 91.96% versus 89.00%, and mIoU 0.9625 versus 0.9488.
  • Per task, the full model reports entry navigation mIoU 0.9162, PR 90.82%, SR 72.73%; lumen traversal mIoU 0.9630, PR 91.46%, SR 100.00%; calculus localization mIoU 0.9816, PR 92.55%, SR 81.82%.

Link