Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning

Source

arXiv

arxiv.org/abs/2607.02513v1

Read the source

Source

Plain-Language Brief

Researchers introduce LACUNA, a testbed for evaluating the precision of unlearning methods in large language models (LLMs). The testbed allows direct evaluation of whether unlearning targets the weights responsible for knowledge storage and finds that existing methods are imprecise and susceptible to resurfacing attacks. A simple gradient-based unlearning method achieves strong erasure and robustness when localization is successful.

What the Document Says

  • The paper introduces LACUNA, a testbed for evaluating the precision of unlearning methods in large language models (LLMs).
  • LACUNA injects personally identifiable information (PII) of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining.
  • It allows direct evaluation of whether unlearning targets the weights responsible for knowledge storage.
  • The paper benchmarks current state-of-the-art (SOTA) unlearning methods using LACUNA and finds that they are highly imprecise and susceptible to resurfacing attacks.
  • When localization is successful, a simple gradient-based unlearning method achieves strong erasure and robustness to resurfacing attacks.
  • The researchers conclude by releasing LACUNA to complement behavioral evaluations and drive further advances in robust, localization-based unlearning.

Key Concepts Mentioned

  • Large Language Models (LLMs)
  • Unlearning
  • Personally Identifiable Information (PII)
  • LACUNA
  • OLMo
  • Resurfacing Attacks

Claims Made by the Document

  • Existing unlearning methods are imprecise and susceptible to resurfacing attacks.
  • When localization is successful, a simple gradient-based unlearning method achieves strong erasure and robustness to resurfacing attacks.

Evidence Used

  • The benchmarking of current SOTA unlearning methods using LACUNA.
  • The findings that existing methods are highly imprecise and susceptible to resurfacing attacks, and the demonstration of strong erasure and robustness when localization is successful.

Methods / Approach

  • Masked continual pretraining
  • Gradient-based unlearning

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLMs memorize sensitive training data, including personally identifiable information (PII), creating a pressing need for reliable post hoc removal methods. Unlearning has emerged as a promising solution, with state-of-the-art(SOTA) methods often following a localize-first, unlearn-second paradigm that targets specific model parameters. However, existing benchmarks evaluate unlearning solely at the output level, leaving open the question of whether unlearning truly erases knowledge from a model's parameters or merely obfuscates it, a concern reinforced by the success of resurfacing attacks. To bridge this gap, we introduce LACUNA: the first unlearning testbed with ground-truth parameter-level localization. LACUNA injects PII of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining, enabling direct evaluation of whether unlearning targets the weights responsible for knowledge storage. We use LACUNA to benchmark current SOTA unlearning methods and find that, despite strong output-level performance, existing methods are highly imprecise and susceptible to resurfacing attacks. We further show that when localization is successful, even a simple gradient-based unlearning method achieves strong erasure and robustness to resurfacing attacks, highlighting the importance of precise unlearning. We release LACUNA to complement behavioral evaluations and drive further advances in robust, localization-based unlearning.