ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers

Source

Source name: arXiv
Source kind: paper
Authors / organization: Mahdi Heidari, Mohammad Mahdi Rahimi, Jaekyun Moon
Venue: arXiv: cs.LG
Published: 2026-07-22
Collected: 2026-07-23
URL: http://arxiv.org/abs/2607.20214v1
PDF: https://arxiv.org/pdf/2607.20214v1
DOI: UNKNOWN

Plain-Language Brief

Researchers propose ELSAA, a method to approximate attention in Transformers more efficiently by using sparse and low-rank branches instead of calculating the full quadratic score matrix. This is done to handle longer input lengths while maintaining sharp token-level interactions and broad contextual mixing.

What the Document Says

The proposed method is called ELSAA.
ELSAA is an efficient low-rank and sparse approximation of attention for Transformers.
ELSAA does not decompose the learned projection or output matrices of the Transformer into sparse and low-rank factors.
After dense projections produce Q, K, V, ELSAA approximates the induced attention score operator itself.
A sparse branch captures selected high-similarity interactions.
A low-rank branch summarizes diffuse global interactions.
ELSAA introduces a denominator-aware fusion term to scale the sparse branch according to its estimated attention mass relative to the low-rank branch.
The goal of ELSAA is to enable longer-context training while preserving both sharp token-level interactions and broad contextual mixing.

Key Concepts Mentioned

ELSAA
Transformers
Attention score matrix
Sparsity
Low-rank/kernel sketches
Quadratic score matrix
Sharp token-level interactions
Broad contextual mixing

Claims Made by the Document

ELSAA is an efficient low-rank and sparse approximation of attention.
ELSAA enables longer-context training.

Evidence Used

The authors propose ELSAA as a solution to the bottleneck caused by the quadratic $N imes N$ attention score matrix in Transformers.
The method approximates the induced attention score operator itself, using sparse and low-rank branches.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

The quadratic $N\times N$ attention score matrix remains a central obstacle to extending Transformers to longer input lengths. Existing efficient attention methods usually reduce this bottleneck by either imposing sparsity, so that each query attends to only a small subset of keys, or by using low-rank/kernel sketches, so that global interactions are compressed into a lower-dimensional representation. We propose \emph{ELSAA}, an efficient low-rank and sparse approximation of attention. Importantly, ELSAA does \emph{not} decompose the learned projection or output matrices of the Transformer into sparse and low-rank factors. Instead, after dense projections produce $Q,K,V$, ELSAA approximates the induced attention score operator itself: a sparse branch captures selected high-similarity interactions, while a low-rank branch summarizes diffuse global interactions. Since the two branches can be normalized over supports with very different denominator mass, ELSAA introduces a denominator-aware fusion term that scales the sparse branch according to its estimated attention mass relative to the low-rank branch. This gives a practical framework for constructing low-rank and sparse attention outputs without materializing the full quadratic score matrix, aiming to enable longer-context training while preserving both sharp token-level interactions and broad contextual mixing.