Source note
AffordTissue: Dense Affordance Prediction for Tool-Action Specific Tissue Interaction
Surgical RoboticsAffordance PredictionVision LanguageDense Heatmap PredictionSafe Manipulation
Summary
AffordTissue predicts dense heatmaps of safe tissue interaction regions for a specific surgical tool and action before contact happens. The paper frames this as a safety layer and policy input for surgical automation in cholecystectomy.
Problem
- Current surgical learning systems and VLA-style models can imitate actions, but they do not show where a tool should safely interact with tissue for a given action.
- Semantic segmentation marks anatomy, not action-specific interaction zones, so it cannot answer where a hook should dissect or where a clipper should clip.
- This matters because clinical deployment needs controllable, inspectable spatial guidance and a way to stop early if the tool moves toward unsafe tissue.
Approach
- The model takes a short text prompt with a surgical triplet
{surgery type, tool type, action type}and a temporal video window of 256 past frames with stride 8, about 10.6 seconds of context. - A frozen SigLIP 2 text encoder embeds the prompt, and a frozen Video Swin Transformer encodes tool motion and tissue dynamics from the video.
- A DiT-style decoder with adaptive layer normalization (AdaLN) fuses the text and video embeddings and predicts per-pixel logits for a dense affordance heatmap on the target frame.
- Training uses manually annotated safe interaction polygons converted into Gaussian-centered heatmaps. The dataset has 15,638 clips from 103 cholecystectomy videos and covers 6 tool-action pairs across 4 instruments: hook, grasper, scissors, and clipper.
- The paper also introduces this dataset as the first benchmark for tissue affordance prediction in this setting.
Results
- On the main benchmark, AffordTissue reports DICE 0.124, PCK@0.05 0.517, PCK@0.1 0.667, HD 79.763 px, and ASSD 20.557 px.
- Against baselines, ASSD improves from 60.184 px for Molmo-VLM to 20.557 px for AffordTissue, and from 81.138 px for SAM3 to 20.557 px. Qwen-VLM (8B) reports 111.271 px ASSD.
- Boundary alignment is much stronger than the baselines: PCK@0.05 is 0.517 for AffordTissue versus 0.128 for SAM3, 0.095 for Molmo-VLM, and 0.031 for Qwen-VLM (8B).
- The paper states the strongest competitor, Molmo-VLM, has 192.76% worse ASSD and 62.34% worse HD relative to AffordTissue.
- Ablations show the language signal is important: removing the language encoder raises ASSD from 20.557 to 43.135 px and HD from 79.763 to 170.482 px.
- Conditioning inputs matter: removing the tool specification increases ASSD to 27.302 px, removing the action increases ASSD to 22.087 px, and removing previous frames increases ASSD to 24.973 px. Replacing AdaLN with cross-attention raises ASSD to 28.736 px.