mllms
7 notes, newest first.
Multimodal Model Diffing for Feature Discovery and Control
Researchers introduce a framework called MMDiff to help identify, control, and audit the internal features of Multimodal Large Language Models (MLLMs). The framework is designed to isolate which features are altered by…
Progressive Multimodal Alignment for Continual Instruction Tuning
The paper introduces Progressive Multimodal Alignment (PMA), a framework that addresses the issue of projector-level forgetting in Multimodal Continual Instruction Tuning (MCIT). PMA allows the projector to adapt…
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers evaluate the ability of six multimodal large language models to understand scientific visualizations (SciVis) using a standardized test with 49 items based on various visualization techniques and tasks. The…
Symbal: Detecting Systematic Misalignments in Model-Generated Captions
Researchers have developed Symbal, a method for detecting systematic misalignments in image-text pairs generated by multimodal large language models (MLLMs). These misalignments occur when the model consistently…
Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference
The paper proposes a method for improving the efficiency of multimodal large language models (MLLMs) by selectively skipping redundant attention and FFN operators during visual-token processing. The method aims to…
Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
Researchers introduce a new method called PEEU (Planning Experience Exploration and Utilization) to improve the task planning capabilities of small, open-source Multimodal Language Learning Models (MLLMs). These models…
AIR: Adaptive Interleaved Reasoning with Code in MLLMs
The paper presents a new approach to enhance multimodal large language models (MLLMs) by giving them the ability to reason adaptively with code, particularly in numerical computation tasks. This is achieved through…
