SARA: Unlocking Multilingual Knowledge in Mixture-of-Experts via Semantically Anchored Routing Alignment

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Tianyu Dong, Yangyang Liu, Jiang Zhou, Xinwei Wu, Xiaohu Zhao, Hao Wang, Heng Liu, Linlong Xu, Longyue Wang, Weihua Luo, Shaolin Zhu, Deyi Xiong
• Venue: arXiv: cs.CL
• Published: 2026-06-24
• Collected: 2026-06-25
• URL: http://arxiv.org/abs/2606.25821v1
• PDF: https://arxiv.org/pdf/2606.25821v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have proposed a framework called SARA (Semantically Anchored Routing Alignment) to improve the performance of multilingual language models in low-resource languages. The framework aligns the routing distribution of multilingual inputs with high-resource semantic anchors, encouraging consistency in expert selection across languages. Experiments on two language models showed that SARA outperformed standard instruction tuning methods.

What the Document Says

• SARA is a framework for improving the performance of multilingual language models
• It addresses the issue of low-resource languages by aligning the routing distribution of multilingual inputs with high-resource semantic anchors
• The goal is to encourage consistency in expert selection across languages
• Experiments were conducted on two language models across five low-resource languages and three benchmarks
• SARA outperformed standard instruction tuning methods, with improvements of +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5-MoE-instruct on Global-MMLU

Key Concepts Mentioned

• SARA
• Multilingual language models
• Low-resource languages
• Semantic anchors
• Expert selection
• Experiment results

Claims Made by the Document

• SARA improves the performance of multilingual language models
• SARA outperforms standard instruction tuning methods

Evidence Used

• The research paper describes SARA and its benefits
• The experiment results are presented in the paper showing superior performance of SARA over standard instruction tuning

Methods / Approach

• Alignment of routing distribution using a symmetric Jensen-Shannon divergence constraint
• Experiments on two language models across five low-resource languages and three benchmarks

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not discuss potential limitations of SARA. It is the agent's responsibility to consider factors such as generalizability, computational complexity, and robustness in various contexts.

Open Questions

• How generalizable is SARA across different types of multilingual language models?
• What is the computational complexity of implementing SARA in real-world applications?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper provides results on two specific language models, but it's unclear how well these results generalize to other types of multilingual language models. Additionally, the computational complexity of SARA is not discussed in the paper.

Raw Abstract / Excerpt

Sparse Mixture-of-Experts (MoE) architectures have emerged as an increasingly influential paradigm as they offer a strategic balance between parameter scalability and computational efficiency. However, low-resource languages, which suffer from a scarcity of high-quality training data, often have their tokens routed to different experts than those predominantly activated by high-resource inputs, which limits cross-lingual expert sharing. This cross-lingual routing divergence consequently hinders their efficacy in multilingual contexts. To address this issue, we propose SARA (Semantically Anchored Routing Alignment), a framework designed to transfer specialized capabilities from high-resource languages as anchors to low-resource languages. SARA explicitly aligns the routing distribution of multilingual inputs with high-resource semantic anchors using a symmetric Jensen-Shannon (JS) divergence constraint. Unlike traditional distillation methods that operate on output logits, SARA directly aligns the internal routing distributions of MoE layers, encouraging mechanistic consistency in expert selection across languages. We conduct experiments on 2 LLMs across 5 low-resource languages and 3 benchmarks. Experiment results demonstrate that SARA outperforms standard instruction tuning, e.g., +0.8% on Qwen3-30B-A3B and +1.2% on Phi-3.5-MoE-instruct on Global-MMLU. Further analyses show that SARA effectively addresses performance bottlenecks in low-resource languages, providing a scalable pathway to enhance multilingual capabilities in sparse architectures.