Source note
2D or 3D: Who Governs Salience in VLA Models? -- Tri-Stage Token Pruning Framework with Modality Salience Awareness
Summary
This paper studies token pruning for vision-language-action models that use both 2D images and 3D point clouds. The main claim is that pruning should follow how 2D and 3D salience changes across preprocessing, semantic reasoning, and action rollout, which gives faster inference with small accuracy loss.
Problem
- Multi-visual-modal VLA models add 3D tokens to 2D inputs, which raises sequence length and slows inference. The paper says these models often run at 3 to 5 Hz, below the roughly 20 to 30 Hz needed for real-time control.
- Existing token pruning methods were built for 2D-only VLA models, so they miss how useful 2D and 3D tokens differ by modality, semantic region, and time step.
- In robot control, pruning the wrong tokens can hurt task success. The paper shows naive pruning can cause large drops, such as 55.00% to 6.67% on Close Box when pruning 2D tokens at 50%.
Approach
- The method builds a tri-stage token pruning framework around the three stages of MVLA inference: data preprocessing, semantic synthesis, and action iteration.
- In stage 1, it measures modality salience from the model's final-layer features using L1 norms and computes separate salience scores for 2D and 3D tokens. These scores set different pruning thresholds for the two modalities.
- In stage 2, it groups patches into semantic sets such as background, robot, and object using attention-based clustering, then measures how much 2D and 3D matter inside each set. The paper also decomposes 3D attention into overlapping and unique parts to estimate the unique value of 3D tokens.
- In stage 3, it tracks how modality salience changes over action steps and adds temporal segmentation plus salience prediction so pruning can adapt during execution.
- The final pruning policy combines these three signals to choose which 2D and 3D tokens to keep.
Results
- The abstract claims up to 2.55× inference speedup with minimal accuracy loss and only 5.8% overhead.
- In the paper's stage-1 analysis on RLBench tasks with the MLA model, pruning 3D tokens is often less harmful than pruning 2D tokens, which supports modality-aware pruning. Example: on Close Box, baseline success rate is 55.00%; with 50% 2D pruning it drops to 6.67%; with 50% 3D pruning it is 40.00%.
- Some tasks improve after 3D pruning in the naive study. Examples at 50% pruning: Close Fridge goes from 56.66% to 70.00%, Close Laptop from 80.00% to 90.00%, and Sweep Dustpan from 66.67% to 96.67%.
- The stage-1 salience metric reports much higher 2D salience than 3D salience across listed tasks, with examples such as 90.16% vs 9.84% on Close Box, 81.47% vs 18.53% on Close Fridge, and 90.38% vs 9.62% on Close Laptop.
- For stage 2 and stage 3, the excerpt gives qualitative claims: 3D salience is higher than 2D in robot and object semantic regions, and modality salience changes over time during manipulation. The excerpt does not provide full end-to-end benchmark tables for these stages beyond the abstract-level speedup claim.