Multimodal Model Diffing for Feature Discovery and Control

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Hunar Batra, Lachin Naghashyar, Ashkan Khakzar, Philip Torr, Christian Schroeder de Witt, Constantin Venhoff, Ronald Clark
• Venue: arXiv: cs.CV
• Published: 2026-08-10
• Collected: 2026-08-11
• URL: http://arxiv.org/abs/2608.09928v1
• PDF: https://arxiv.org/pdf/2608.09928v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers introduce a framework called MMDiff to help identify, control, and audit the internal features of Multimodal Large Language Models (MLLMs). The framework is designed to isolate which features are altered by multimodal training, detect task-specific features, and control these features for improved safety and capability.

What the Document Says

• The research presents MMDiff, a framework that identifies and controls the internal features of Multimodal Large Language Models (MLLMs).
• MMDiff trains multimodal sparse autoencoders (SAEs) to serve as feature-level interfaces for discovering and controlling multimodal behavior.
• The framework supports three uses: feature isolation, task-specific feature detection, and feature-level control.
• It is tested on three MLLM families: LLaVA-MORE, PaliGemma 2, and InternVL3.5.
• Evaluation focuses on visual-spatial understanding, multimodal safety, and OCR.
• The removal of discovered features selectively degrades target behaviors by an average of 12% on spatial tasks and 17% on OCR.
• Removal of these features reduces attack success rate by 24% on multimodal safety attacks.
• No impact on VQA performance is reported.
• Steering the discovered features improves spatial and OCR accuracy by an average of +3.6% and +1.8% respectively, compared to a standard single-layer steering baseline.

Key Concepts Mentioned

• Multimodal Large Language Models (MLLMs)
• sparse autoencoders (SAEs)
• feature isolation
• task-specific feature detection
• feature-level control
• LLaVA-MORE
• PaliGemma 2
• InternVL3.5

Claims Made by the Document

• Multimodal SAEs can serve as interpretability tools.
• Removal of discovered features degrades target behaviors.
• Removal of these features reduces attack success rate.
• Steering the discovered features improves accuracy.

Evidence Used

• Training multimodal SAEs for three MLLM families and evaluation on various tasks.
• Results showing the average degradation and improvement in performance due to feature removal and steering respectively.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control. We introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses: (i) feature isolation, by diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training; (ii) task-specific feature detection, via per-token contrastive firing analysis that isolates causal features; and (iii) feature-level control, by causally removing or steering the discovered feature directions. We train multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding, multimodal safety, and OCR. MMDiff discovers sparse, causally specific features whose removal selectively degrades target behaviors by an average of 12% on spatial tasks and 17% on OCR, and reduces attack success rate by 24% on multimodal safety attacks, with no impact on VQA performance. Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average over a standard single-layer steering baseline. These results show that multimodal SAEs can serve not only as interpretability tools, but as mechanisms for auditing, steering, and controlling MLLMs behavior toward safer and more capable generations.