Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Jiahe Fan, Yinghao Hou, Si Chen, Aiyuan Zhang, Hong Xie, Defu Lian
• Venue: arXiv: cs.AI
• Published: 2026-07-20
• Collected: 2026-07-21
• URL: http://arxiv.org/abs/2607.18026v1
• PDF: https://arxiv.org/pdf/2607.18026v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers propose a simple method to merge large language models with significantly different parameters without training or alignment by using dimensional adaptation and ratio-controlled interpolation. The method, tested on various tasks, shows that expanding the smaller model into the larger parameter space and interpolating with carefully controlled ratios can improve performance over strong source checkpoints.

What the Document Says

• The paper discusses a method to merge large language models (LLMs) with substantially different parameter spaces without training or semantic alignment.
• The method involves training-free dimensional adaptation followed by ratio-controlled interpolation.
• Union-style merging expands the smaller model into the larger parameter space, while intersection-style merging truncates the larger model into the smaller parameter space.
• Testing was done on Qwen-family model pairs and benchmarks covering various tasks such as mathematical reasoning, code generation, language understanding, commonsense reasoning, knowledge, and instruction following.
• Deterministic expansion largely preserves the source model function, and small-ratio interpolation can improve over strong source checkpoints by transferring complementary capabilities.
• Near-balanced interpolation often collapses, and task-level results reveal a seesaw effect where gains on some capabilities coexist with regressions on others.

Key Concepts Mentioned

• large language models
• parameter spaces
• dimensional adaptation
• ratio-controlled interpolation
• union-style merging
• intersection-style merging

Claims Made by the Document

• Direct weighted averaging of large language models can be effective for heterogeneous fusion.
• Expanding the smaller model into the larger parameter space largely preserves the source model function.

Evidence Used

• Results from experiments on various tasks show that interpolation can improve over strong source checkpoints by transferring complementary capabilities.
• The observation that near-balanced interpolation often collapses and results reveal a seesaw effect between gains and regressions.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment? Existing heterogeneous fusion methods typically introduce distillation, adapters, learned latent spaces, routing, or feature alignment, leaving open whether a simpler recipe can work for genuinely different billion-parameter checkpoints. We revisit this counterintuitive question through training-free dimensional adaptation followed by ratio-controlled interpolation. In union-style merging, we expand the smaller model into the larger parameter space; in intersection-style merging, we truncate the larger model into the smaller parameter space. Across Qwen-family model pairs and benchmarks covering mathematical reasoning, code generation, language understanding, commonsense reasoning, knowledge, and instruction following, deterministic expansion largely preserves the source model function, and small-ratio interpolation can improve over strong source checkpoints by transferring complementary capabilities. However, near-balanced interpolation often collapses, and task-level results reveal a seesaw effect in which gains on some capabilities coexist with regressions on others. These results show that simple parameter averaging, when paired with lightweight dimensional adaptation and carefully controlled ratios, is a surprisingly strong baseline for heterogeneous LLM merging, suggesting that the limits of direct weighted fusion may also bound what more complex heterogeneous merging methods can achieve at scale.