Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SLORR: Simple and Efficient In-Training Low-Rank Regularization

Source

arXiv

arxiv.org/abs/2607.08754v1

Read the source

Source

Plain-Language Brief

The paper introduces SLORR, a simple and efficient in-training low-rank regularization framework for neural networks. It aims to improve compressibility without significant accuracy loss, while avoiding the limitations of existing methods such as requiring large SVDs, modifying model architecture, or relying on stateful cached quantities. The authors provide two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR is evaluated on various models for image classification and language modeling, showing improved compressibility with minimal training overhead.

What the Document Says

  • The paper proposes SLORR, a framework for low-rank regularization during training of neural networks.
  • SLORR aims to improve compressibility without significant accuracy loss.
  • Existing methods have limitations such as requiring large SVDs, modifying model architecture, or relying on stateful cached quantities.
  • SLORR uses GPU-friendly approximations for the forward and backward passes of the regularizers.
  • The authors provide approximation guarantees for these approximations.
  • Two main variants of SLORR are based on the Hoyer sparsity metric and the nuclear norm.

Key Concepts Mentioned

  • Low-rank factorization
  • Neural networks
  • Compressibility
  • Accuracy loss
  • Training overhead

Claims Made by the Document

  • SLORR improves compressibility without significant accuracy loss.
  • SLORR introduces less than 8% training overhead.

Evidence Used

  • Evaluations on ImageNet-1K and LLM pretraining at various scales.

Methods / Approach

  • SLORR regularizes original weight matrices with GPU-friendly approximations for the forward and backward passes.
  • Two main variants are based on the Hoyer sparsity metric and the nuclear norm.

Limitations Stated by the Authors

  • The paper does not discuss the potential impact of SLORR on smaller models or other types of tasks beyond image classification and language modeling.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The limitations agent acknowledges that the effectiveness of SLORR in other domains, such as reinforcement learning or natural language understanding, is not discussed in the paper.
  • The limitations agent also notes that the long-term impact of SLORR on model performance and computational efficiency might be affected by future advancements in neural network architectures or optimization methods.

Open Questions

  • How does SLORR perform on smaller models?
  • How does SLORR perform on other types of tasks beyond image classification and language modeling?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper evaluates SLORR on short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18. However, it does not provide explicit information about the impact of SLORR on smaller models.
  • The paper focuses on image classification and language modeling tasks, but does not discuss its application to other types of tasks.

Raw Abstract / Excerpt

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.