Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

Source

arXiv

arxiv.org/abs/2608.09742v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Xinyi Xu, Bingnan Xiao, Shuang Qin, Gang Feng, Tony Q. S. Quek
  • Venue: arXiv: cs.LG
  • Published: 2026-08-10
  • Collected: 2026-08-11
  • URL: http://arxiv.org/abs/2608.09742v1
  • PDF: https://arxiv.org/pdf/2608.09742v1
  • DOI: UNKNOWN
  • Registry group: academic_apis_and_preprints
  • Evidence role: canonical_academic_index
  • Language: en
  • Country / region: UNKNOWN

Plain-Language Brief

Researchers propose a new method for fine-tuning large language models in a federated learning paradigm using Low-rank adaptation (LoRA). They investigate two strategies, Share-A/Local-B and Share-B/Local-A, and reveal that the preferred strategy is the one with the smaller aggregate residual across clients. To make an informed decision about which strategy to use before training, they introduce a Rank-Aware Shared-Subspace Sufficiency (RSS) metric. The experiments show the effectiveness of RSS and the superior performance of their proposed method, Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), across various tasks, data distributions, LoRA ranks, and participation settings.

What the Document Says

  • Low-rank adaptation (LoRA) is a method for fine-tuning large language models.
  • Two strategies are studied: Share-A/Local-B and Share-B/Local-A.
  • The preferred strategy depends on the one with the smaller aggregate residual across clients.
  • Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA) is proposed, which selects the sharing side before training to enhance fine-tuning performance.
  • A Rank-Aware Shared-Subspace Sufficiency (RSS) metric is introduced for adaptive factor selection before training.
  • The effectiveness of RSS and the superior performance of FedAS-LoRA are confirmed across various tasks, data distributions, LoRA ranks, and participation settings.

Key Concepts Mentioned

  • Low-rank adaptation (LoRA)
  • fine-tuning
  • federated learning paradigm
  • Share-A/Local-B
  • Share-B/Local-A
  • aggregate residual
  • Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA)
  • Rank-Aware Shared-Subspace Sufficiency (RSS)

Claims Made by the Document

  • The preferred strategy for LoRA fine-tuning depends on the one with the smaller aggregate residual across clients.
  • RSS is effective and FedAS-LoRA performs superiorly across various tasks, data distributions, LoRA ranks, and participation settings.

Evidence Used

  • With a least-squares surrogate, they reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-$r$ input-side space.
  • The two strategies therefore incur different projection residuals.
  • Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Methods / Approach

  • Low-rank adaptation (LoRA)
  • least-squares surrogate
  • Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA)
  • Rank-Aware Shared-Subspace Sufficiency (RSS)

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study focuses on specific strategies and a particular method for fine-tuning large language models, limiting the scope of its applicability.

Open Questions

Potential future research could explore other strategies or methods for fine-tuning large language models in federated learning paradigms.

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper discusses the importance of selecting the right strategy (Share-A/Local-B or Share-B/Local-A) before training to improve the performance of Low-rank adaptation (LoRA). It proposes a new method, Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), that uses a Rank-Aware Shared-Subspace Sufficiency (RSS) metric for adaptive factor selection before training. The experiments confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Raw Abstract / Excerpt

Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., $A$ and $B$, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B), or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-$r$ input-side space, whereas Share-B/Local-A requires a common rank-$r$ output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-$r$ input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.