FIELD NOTES TOPIC
heavy-tailed noise
1 note, newest first.
Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
The authors question if AdamW, a commonly used optimizer for training large language models, is effective under heavy-tailed noise as its theory only applies to finite-variance regimes. They mention that sign-based…
