Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

Source

arXiv

arxiv.org/abs/2607.21300v1

Read the source

Source

Plain-Language Brief

Researchers propose a new benchmark, FAIRGET, to evaluate the effectiveness of machine unlearning in multimodal large language models under unbalanced, realistic scenarios. They also introduce FAUN, an unlearning algorithm designed for MLLMs that preserves fairness while forgetting unlearning data.

What the Document Says

  • Machine unlearning is used to remove personal data from trained models in compliance with AI regulations.
  • Prior works evaluate unlearning effectiveness in multimodal large language models (MLLMs) by fine-tuning models on fictitious identities, simulating unlearning requests on subsets of these IDs which are typically uniformly distributed.
  • However, in realistic scenarios, people from different demographic groups may request to be unlearned at different frequencies, potentially leading to biased behaviors.
  • The paper proposes FAIRGET, a Visual Question Answering benchmark that evaluates unlearning under unbalanced, realistic forget requests.
  • FAIRGET simulates multiple scenarios ranging from simple to challenging settings that lead to biased unlearned models if fairness is not accounted for.
  • The authors also propose FAUN, an unlearning algorithm for MLLMs that forgets unlearning data while preserving model fairness.
  • FAUN uses a bias-aware activation steering mechanism to unlearn identities while accounting for the unbalanced nature of the forget data.
  • Experiments on FAIRGET and the established FIUBench demonstrate FAUN's superiority both in unlearning quality and fairness.

Key Concepts Mentioned

  • Machine unlearning
  • Multimodal large language models (MLLMs)
  • Fairness
  • FAIRGET
  • FAUN
  • Visual Question Answering benchmark
  • Bias-aware activation steering mechanism

Claims Made by the Document

  • Current unlearning evaluation methods are not suitable for realistic scenarios.
  • FAIRGET is the first Visual Question Answering benchmark to evaluate unlearning under unbalanced, realistic forget requests.
  • FAUN is the first unlearning algorithm for MLLMs that preserves fairness while forgetting unlearning data.
  • FAUN's experiments on FAIRGET and FIUBench demonstrate its superiority both in unlearning quality and fairness.

Evidence Used

  • The paper describes the current methods for evaluating unlearning effectiveness in MLLMs, which are based on fictitious identities with uniformly distributed forget requests.
  • The authors describe FAIRGET as a new benchmark designed to fill this gap by simulating multiple realistic scenarios.
  • FAUN is presented as an algorithm that exploits a bias-aware activation steering mechanism to address the unbalanced nature of forget data in MLLMs.
  • Experiments on both FAIRGET and FIUBench are mentioned to demonstrate the effectiveness of FAUN.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations. To evaluate unlearning effectiveness in multimodal large language models (MLLMs), prior works fine-tune models on fictitious identities, simulating unlearning requests on subsets of these IDs, which are typically uniformly distributed. However, in realistic scenarios, people from different demographic groups may request to be unlearned at different frequencies, potentially altering the model's internal beliefs for these groups and leading to biased behaviors. To fill this gap, we propose FAIRGET, the first Visual Question Answering benchmark that evaluates unlearning under unbalanced, realistic, forget requests. These requests are designed to simulate multiple realistic scenarios, ranging from simple to challenging settings, that lead to biased unlearned models if fairness is not accounted for. Additionally, we propose FAUN, the first unlearning algorithm for MLLMs that forgets unlearning data while preserving model fairness. FAUN exploits a bias-aware activation steering mechanism to unlearn identities while accounting for the unbalanced nature of the forget data. Experiments on FAIRGET and the established FIUBench demonstrate our method's superiority both in unlearning quality and fairness.