FIELD NOTES TOPIC
plasticity loss
2 notes, newest first.
Can Scale Save Us From Plasticity Loss in Large Language Models?
Researchers studied plasticity loss, a challenge in creating neural networks capable of continuous learning, in modern transformer-based language models (LLMs). They found evidence of plasticity loss across various…
Can Scale Save Us From Plasticity Loss in Large Language Models?
Researchers study plasticity loss in GPT-style transformer models trained on a multilingual continual learning problem. They find evidence of plasticity loss across different sizes of models and under stationary…
