Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Dingzhi Yu, Hongyi Tao, Yuanyu Wan, Luo Luo, Lijun Zhang
• Venue: arXiv: cs.LG
• Published: 2026-06-22
• Collected: 2026-06-23
• URL: http://arxiv.org/abs/2606.23676v1
• PDF: https://arxiv.org/pdf/2606.23676v1
• DOI: UNKNOWN

Plain-Language Brief

The authors question if AdamW, a commonly used optimizer for training large language models, is effective under heavy-tailed noise as its theory only applies to finite-variance regimes. They mention that sign-based optimizers like Lion and Muon have shown promising results in heavy-tailed scenarios but no rigorous convergence theory exists for AdamW. The authors propose this as an open problem, present a positive weighted-metric benchmark, and discuss how denominator memory can hide large gradients.

What the Document Says

• AdamW is widely used to train large language models (LLMs), but its theory only applies to finite-variance regimes.
• Empirical evidence suggests that stochastic gradient noise in LLM pretraining is typically heavy-tailed.
• Sign-based optimizers like Lion and Muon have demonstrated sharp heavy-tailed rates, while AdaGrad can also converge under heavy-tailed noise.
• No rigorous convergence theory for AdamW has been established in the heavy-tailed regime.
• The authors propose this as an open problem: Can AdamW converge under heavy-tailed assumptions or is its second-moment accumulator a genuine obstruction?
• They present a positive weighted-metric benchmark and discuss how denominator memory can hide large gradients.

Key Concepts Mentioned

• AdamW
• optimizers
• large language models
• heavy-tailed noise
• finite-variance regimes
• sign-based optimizers
• Lion
• Muon
• AdaGrad
• convergence theory

Claims Made by the Document

• AdamW's effectiveness under heavy-tailed noise is questionable due to its reliance on finite-variance regimes.
• Sign-based optimizers like Lion and Muon perform well in heavy-tailed scenarios.
• AdaGrad can converge under heavy-tailed noise.

Evidence Used

• The authors state that the theory behind AdamW is limited to finite-variance regimes. They also mention empirical evidence indicating heavy-tailed stochastic gradient noise in LLM pretraining.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction? We formulate this as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism showing how denominator memory can hide large gradients.