FIELD NOTES TOPIC
optimizers
2 notes, newest first.
Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
The researchers compare the performance of various matrix-structured optimizers (including Muon, SOAP, and SOAP-Muon) to Adam in training machine learning interatomic potentials (MLIPs). They found that these…
Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?
The authors question if AdamW, a commonly used optimizer for training large language models, is effective under heavy-tailed noise as its theory only applies to finite-variance regimes. They mention that sign-based…
