Source note
AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning
Vision Language ActionCatastrophic ForgettingGradient ProjectionRobot Fine TuningKnowledge Preservation
Summary
AEGIS is a fine-tuning method for vision-language-action models that keeps a pre-trained vision-language model's visual reasoning ability while still letting continuous action gradients update the backbone. It does this by detecting, layer by layer, when action-training gradients would push the model away from its pre-trained activation patterns and removing only that conflicting component.
Problem
- Vision-language models are pre-trained with cross-entropy gradients spread across many semantic directions, while robot action fine-tuning injects high-magnitude MSE gradients concentrated in a narrow control subspace.
- This mismatch causes catastrophic forgetting: the paper reports that naive direct MSE fine-tuning causes VQA holdout loss to degrade steadily within 1,500 training steps.
- Existing defenses have clear costs: stop-gradient preserves VQA by blocking action gradients, LoRA slows but does not stop VQA erosion, and mixed-batch VQA co-training can use up to 50% VQA data during robot training, adding compute and data overhead.
Approach
- AEGIS pre-computes a static activation anchor from masked VQA forward passes over all 26 transformer layers, storing per-layer Gaussian mean and variance statistics.
- During robot fine-tuning, it measures how current layer activations drift from that anchor with a per-layer Wasserstein-2 distance using mean shift and standard-deviation mismatch.
- It runs two backward passes on the same graph: one for the action loss and one for the anchor-restoration objective, producing a task gradient and an anchor gradient.
- For each transformer layer, it checks whether the action gradient conflicts with the anchor-restoration direction. If the dot product is negative, it projects out the conflicting component with a Gram-Schmidt step; if not, it leaves the gradient unchanged.
- The method changes only the backward pass. The paper says the forward pass, architecture, and task losses stay unchanged, and training needs no replay buffer, no co-training batches, and no discrete action tokens.
Results
- The paper claims naive direct MSE fine-tuning causes a significant, steady increase in VQA holdout loss within 1,500 steps.
- The stop-gradient baseline keeps VQA loss roughly constant, according to the text, but blocks continuous supervision from the action expert.
- The LoRA baseline uses rank 16 and alpha 32 on LLM layers and still shows a steady increase in VQA loss, though slower than naive fine-tuning.
- AEGIS builds its anchor in about 5 minutes on a single GPU using 3,000 VQAv2 samples.
- The projection removes less than 1% of gradient energy on average, according to the abstract.
- The excerpt does not provide full benchmark tables or end-task robot success numbers, so there is no quantitative comparison here for manipulation performance, dataset-wide VQA scores, or baselines beyond the concrete claims above.