Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
7 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on mllms
7 notes
Multimodal Model Diffing for Feature Discovery and Control
Researchers introduce a framework called MMDiff to help identify, control, and audit the internal features of Multimodal Large Language Models (MLLMs). The framework is designed to isolate which features are altered by…
Progressive Multimodal Alignment for Continual Instruction Tuning
The paper introduces Progressive Multimodal Alignment (PMA), a framework that addresses the issue of projector-level forgetting in Multimodal Continual Instruction Tuning (MCIT). PMA allows the projector to adapt…
Benchmarking Multimodal Large Language Models for Scientific Visualization Literacy
Researchers evaluate the ability of six multimodal large language models to understand scientific visualizations (SciVis) using a standardized test with 49 items based on various visualization techniques and tasks. The…
Symbal: Detecting Systematic Misalignments in Model-Generated Captions
Researchers have developed Symbal, a method for detecting systematic misalignments in image-text pairs generated by multimodal large language models (MLLMs). These misalignments occur when the model consistently…
Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference
The paper proposes a method for improving the efficiency of multimodal large language models (MLLMs) by selectively skipping redundant attention and FFN operators during visual-token processing. The method aims to…
Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
Researchers introduce a new method called PEEU (Planning Experience Exploration and Utilization) to improve the task planning capabilities of small, open-source Multimodal Language Learning Models (MLLMs). These models…
AIR: Adaptive Interleaved Reasoning with Code in MLLMs
The paper presents a new approach to enhance multimodal large language models (MLLMs) by giving them the ability to reason adaptively with code, particularly in numerical computation tasks. This is achieved through…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
