Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Can Scale Save Us From Plasticity Loss in Large Language Models?

Source

arXiv

arxiv.org/abs/2606.24752v1

Read the source

Source

Plain-Language Brief

Researchers studied plasticity loss, a challenge in creating neural networks capable of continuous learning, in modern transformer-based language models (LLMs). They found evidence of plasticity loss across various sizes of GPT-style Transformer models trained on a multilingual continual learning problem. The onset of plasticity loss followed a predictable scaling law, growing sublinearly with model size. Larger models might delay the measurable effects of plasticity loss but may not completely prevent it.

What the Document Says

  • The study investigates plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem.
  • Evidence of plasticity loss is found across models ranging from 5M to 314M non-embedding parameters as measured by deterioration on a held-out Vietnamese probing task.
  • The onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size.
  • Larger models might delay the measurable effects of plasticity loss but may not completely prevent it.
  • Plasticity loss is also found under stationary multilingual training, which contradicts the notion that the phenomenon is exclusive to continual learning with abrupt task changes.

Key Concepts Mentioned

  • plasticity loss
  • GPT-style Transformer models
  • multilingual continual learning problem
  • deterioration
  • scaling law

Claims Made by the Document

  • Plasticity loss is a problem in the modern transformer-based LLM paradigm.
  • Larger models might delay but not completely prevent plasticity loss.
  • Plasticity loss occurs under stationary multilingual training.

Evidence Used

  • Evidence from various sizes of GPT-style Transformer models.
  • Deterioration on a held-out Vietnamese probing task.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains. To determine whether loss of plasticity remains a problem in the modern transformer-based LLM paradigm, we study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem. Consistent with prior work, we find evidence of plasticity loss across models ranging from 5M to 314M non-embedding parameters, as measured by deterioration on a held-out Vietnamese probing task. We further find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size. These results suggest that larger models may delay the measurable effects of plasticity loss, but that increasing parameter count alone is likely to be insufficient to completely prevent it. We also find evidence of plasticity loss under stationary multilingual training, challenging the view that the phenomenon is exclusive to continual learning with abrupt task changes. Overall, our results suggest that even large Transformer language models trained on natural-language will eventually lose the ability to efficiently adapt to new data after sufficiently long training, in both continual and stationary settings.