SLORR: Simple and Efficient In-Training Low-Rank Regularization

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: David González-Martínez, Shiwei Liu
• Venue: arXiv: cs.LG
• Published: 2026-07-09
• Collected: 2026-07-10
• URL: http://arxiv.org/abs/2607.08754v1
• PDF: https://arxiv.org/pdf/2607.08754v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces SLORR, a simple and efficient low-rank regularization method for compressing neural networks during training. It addresses the limitations of existing methods by being stateless, architecture-preserving, and using GPU-friendly approximations for the forward and backward passes. The authors evaluate SLORR on various models and tasks, showing that it induces compressibility with minimal training overhead.

What the Document Says

• SLORR is a low-rank regularization method for neural networks
• It is simple, stateless, and architecture-preserving
• Two variants of SLORR are based on the Hoyer sparsity metric and nuclear norm
• SLORR directly regularizes original weight matrices using GPU-friendly approximations
• Approximation guarantees are provided for the forward and backward passes
• SLORR was evaluated on ImageNet-1K and language model pretraining tasks

Key Concepts Mentioned

• Low-rank factorization
• Neural networks
• Compressibility
• Training overhead
• GPU-friendly approximations
• Hoyer sparsity metric
• Nuclear norm

Claims Made by the Document

• SLORR improves compressibility
• It introduces minimal training overhead
• SLORR-trained compressed models preserve performance better than unregularized models

Evidence Used

• Results from ImageNet-1K and language model pretraining tasks are provided

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.