FIELD NOTES TOPIC
longer context training
1 note, newest first.
ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
Researchers propose ELSAA, a method to approximate attention in Transformers more efficiently by using sparse and low-rank branches instead of calculating the full quadratic score matrix. This is done to handle longer…
