Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SLORR: Simple and Efficient In-Training Low-Rank Regularization

Source

arXiv

arxiv.org/abs/2607.08754v1

Read the source

Source

Plain-Language Brief

The paper introduces SLORR, a simple and efficient in-training low-rank regularization framework for compressing neural networks without significant accuracy loss or relying on stateful cached quantities. The authors provide two main variants of SLORR based on the Hoyer sparsity metric and the nuclear norm, which can be implemented with GPU-friendly approximations for the forward and backward passes. The authors evaluate SLORR on various models in ImageNet-1K continued training and LLM pretraining, showing that it induces compressibility while adding minimal training overhead.

What the Document Says

  • The paper presents a low-rank regularization framework called SLORR for neural networks.
  • SLORR is designed to be simple, stateless, and architecture-preserving.
  • It has two main variants based on the Hoyer sparsity metric and the nuclear norm.
  • SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes.
  • The authors provide approximation guarantees for these approximations.
  • SLORR is evaluated on ImageNet-1K continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18.
  • SLORR induces compressibility while introducing less than 8% training overhead.

Key Concepts Mentioned

  • Low-rank factorization
  • Neural networks
  • Compressibility
  • In-training low-rank regularization
  • Hoyer sparsity metric
  • Nuclear norm
  • ImageNet-1K
  • ResNet
  • ViT
  • Training overhead

Claims Made by the Document

  • SLORR improves compressibility of modern models.
  • Existing low-rank regularizers have limitations like requiring SVDs of large weight matrices, modifying the model architecture, or relying on stateful cached quantities.
  • SLORR induces less than 8% training overhead.

Evidence Used

  • The paper provides experimental results demonstrating SLORR's effectiveness in terms of compressibility and training overhead.

Methods / Approach

  • Low-rank regularization using SLORR
  • GPU-friendly approximations for the forward and backward passes

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not discuss any potential limitations of the proposed method from the agent's perspective.

Open Questions

  • Can SLORR be applied to other datasets or tasks beyond those mentioned in the paper?
  • How does SLORR perform compared to other low-rank regularization methods on various neural network architectures?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on ImageNet-1K and LLM pretraining, so it's unclear how well SLORR would perform on other datasets or tasks.
  • It's not explicitly stated whether SLORR outperforms other low-rank regularization methods in terms of compressibility and training overhead.

Raw Abstract / Excerpt

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.