Source note
UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving
Summary
UniDriveVLA is a unified vision-language-action model for autonomous driving that separates understanding, perception, and planning into different transformer experts. The paper claims this avoids interference between spatial perception and language reasoning and reaches state-of-the-art results on nuScenes open-loop evaluation and Bench2Drive closed-loop evaluation.
Problem
- Existing driving VLA models face a tradeoff: plain 2D vision-language models keep language reasoning but miss spatial structure needed for driving.
- Adding 3D or spatial tokens into shared model parameters can hurt the pre-trained vision-language model's reasoning ability through representation interference.
- This matters because autonomous driving needs both semantic understanding and precise spatial perception for safe planning.
Approach
- The model uses a Mixture-of-Transformers design with three experts: a driving-understanding expert, a scene-perception expert, and an action-planning expert.
- Tokens for understanding, perception, and action go through separate expert-specific projections, so the three functions do not compete in the same parameter subspace.
- A masked joint attention layer allows controlled information flow: understanding tokens keep causal masking, perception tokens can read earlier understanding tokens, and action tokens can read both semantic and spatial context.
- The perception branch uses a sparse query-based decoder built from multi-scale 2D visual features instead of dense BEV or dense 3D grids; it predicts 3D detection, mapping, ego status, motion, and occupancy.
- Training happens in three stages: multimodal pretraining for semantic ability, joint training for language/perception/planning with LoRA and reduced VLM learning rate, then expert fine-tuning with the VLM frozen and an added motion objective.
Results
- The paper claims state-of-the-art performance on nuScenes for open-loop evaluation and on Bench2Drive for closed-loop evaluation.
- Reported benchmark metrics shown in the excerpt include Avg. L2 for open-loop planning and Driving Score / Success Rate / Efficiency / Comfortness for closed-loop evaluation, but the UniDriveVLA row is not visible in the provided text, so its exact numbers are not available here.
- The excerpt gives baseline closed-loop numbers on Bench2Drive for prior VLA methods: AutoVLA reaches 78.84 Driving Score and 57.73% Success Rate; SimLingo reaches 85.94 Driving Score and 66.82% Success Rate; R2SE reaches 86.28 Driving Score and 69.54% Success Rate.
- A non-VLA baseline, AD-MLP, shows 3.64 Avg. L2 on open-loop and 18.05 Driving Score with 0.00% Success Rate on closed-loop Bench2Drive.
- The paper also claims strong performance across 3D detection, online mapping, motion forecasting, and driving-oriented VQA, plus evaluation on DriveBench and several general VQA benchmarks, but the excerpt does not provide those task-specific scores.
- Figure 2 reports a qualitative mechanism result: in a shared-weight decoder, cosine similarity between language and perception tokens rises toward 1, while the MoT design keeps similarity low, which the authors use as evidence of reduced feature collapse and better task separation.