FIELD NOTES TOPIC
rank preservation
1 note, newest first.
Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
The research investigates how the design of Transformer feedforward block architecture affects the preservation of 'rank' (a mathematical concept related to the expressiveness of a network) across depth during…
