Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearning

Source

arXiv

arxiv.org/abs/2607.21300v1

Read the source

Source

Plain-Language Brief

Researchers propose FAIRGET, a Visual Question Answering benchmark to evaluate the effectiveness of unlearning (removing personal data from trained models) in multimodal large language models under unbalanced, realistic scenarios. They also introduce FAUN, an unlearning algorithm that forgets unlearning data while maintaining model fairness.

What the Document Says

  • Machine unlearning is a tool for removing personal data from trained models to comply with AI regulations.
  • Prior works fine-tune models on fictitious identities, simulating unlearning requests on subsets of these IDs which are typically uniformly distributed.
  • However, in realistic scenarios, people from different demographic groups may request to be unlearned at different frequencies.
  • The proposed benchmark, FAIRGET, evaluates unlearning under unbalanced, realistic forget requests.
  • FAIRGET simulates multiple realistic scenarios, ranging from simple to challenging settings.
  • FAUN is an unlearning algorithm for MLLMs that forgets unlearning data while preserving model fairness.
  • FAUN uses a bias-aware activation steering mechanism to unlearn identities while accounting for the unbalanced nature of the forget data.
  • Experiments on FAIRGET and FIUBench demonstrate FAUN's superiority in unlearning quality and fairness.

Key Concepts Mentioned

  • Machine unlearning
  • Multimodal large language models (MLLMs)
  • Fairness
  • FAIRGET
  • FAUN
  • Bias-aware activation steering mechanism
  • FIUBench

Claims Made by the Document

  • Prior works fine-tune models on fictitious identities under uniform distribution.
  • In realistic scenarios, people may request to be unlearned differently across demographic groups.
  • FAIRGET is the first Visual Question Answering benchmark to evaluate unlearning under unbalanced, realistic forget requests.
  • FAUN is the first unlearning algorithm for MLLMs that accounts for unbalanced forget data.
  • Experiments on FAIRGET and FIUBench demonstrate FAUN's superiority.

Evidence Used

  • Text: 'To evaluate unlearning effectiveness in multimodal large language models (MLLMs), prior works fine-tune models on fictitious identities, simulating unlearning requests on subsets of these IDs, which are typically uniformly distributed.'
  • Text: 'However, in realistic scenarios, people from different demographic groups may request to be unlearned at different frequencies, potentially altering the model's internal beliefs for these groups and leading to biased behaviors.'
  • Text: 'To fill this gap, we propose FAIRGET, the first Visual Question Answering benchmark that evaluates unlearning under unbalanced, realistic forget requests.'
  • Text: 'FAUN exploits a bias-aware activation steering mechanism to unlearn identities while accounting for the unbalanced nature of the forget data.'
  • Text: 'Experiments on FAIRGET and the established FIUBench demonstrate our method's superiority both in unlearning quality and fairness.'

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Machine unlearning has emerged as a tool for removing personal data from trained models to comply with recent AI regulations. To evaluate unlearning effectiveness in multimodal large language models (MLLMs), prior works fine-tune models on fictitious identities, simulating unlearning requests on subsets of these IDs, which are typically uniformly distributed. However, in realistic scenarios, people from different demographic groups may request to be unlearned at different frequencies, potentially altering the model's internal beliefs for these groups and leading to biased behaviors. To fill this gap, we propose FAIRGET, the first Visual Question Answering benchmark that evaluates unlearning under unbalanced, realistic, forget requests. These requests are designed to simulate multiple realistic scenarios, ranging from simple to challenging settings, that lead to biased unlearned models if fairness is not accounted for. Additionally, we propose FAUN, the first unlearning algorithm for MLLMs that forgets unlearning data while preserving model fairness. FAUN exploits a bias-aware activation steering mechanism to unlearn identities while accounting for the unbalanced nature of the forget data. Experiments on FAIRGET and the established FIUBench demonstrate our method's superiority both in unlearning quality and fairness.