HCRMap: Pressure-Aware Hot-Expert Residency Mapping for 3.5D MoE Chiplet Inference

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Yongqin Zhang
• Venue: arXiv: cs.AI
• Published: 2026-07-13
• Collected: 2026-07-14
• URL: http://arxiv.org/abs/2607.11586v1
• PDF: https://arxiv.org/pdf/2607.11586v1
• DOI: UNKNOWN

Plain-Language Brief

The paper proposes HCRMap, a framework for managing hot expert replicas in 3.5D Mixture-of-Experts (MoE) inference systems. The aim is to dynamically place and reuse hot experts across different memory tiers to mitigate communication, memory, and queue bottlenecks. The results show significant reductions in end-to-end latency compared to several other methods.

What the Document Says

• The paper presents HCRMap, a framework for managing hot expert replicas in 3.5D MoE inference systems.
• HCRMap dynamically determines which experts should be promoted, retained, demoted, or evicted based on factors like expert hotness, weight loading cost, migration overhead, and runtime resource pressure.
• It maps routed token groups to suitable resident replicas to mitigate communication, memory, and queue bottlenecks.

Key Concepts Mentioned

• Mixture-of-Experts (MoE) inference
• 3.5D multi-chiplet systems
• hot expert residency mapping
• expert hotness
• communication bottlenecks
• memory bandwidth
• I/O
• execution queues

Claims Made by the Document

• HCRMap reduces end-to-end latency compared to several other methods.
• HCRMap dynamically manages hot expert replicas.

Evidence Used

• Reductions in end-to-end latency by 43.6% and 43.0% over Hydra, 34.5% and 33.1% over MoEntwine, and 46.7% and 46.0% over PIMoE are stated in the results section.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Mixture-of-Experts (MoE) large language models (LLM) activate only a small number of experts during inference, but token routing introduces persistent expert hotness skew: a small set of hot experts continuously receives most tokens, while the remaining experts are lightly loaded. On 3.5D multi-chiplet systems, this skew not only causes compute imbalance but also amplifies pressure on communication, memory bandwidth, I/O, and execution queues. Therefore, the core problem is not simply to reduce token movement, but to dynamically place and reuse hot expert replicas across different memory tiers. This paper proposes HCRMap, a hot expert residency mapping framework for pressure-aware expert replica management in 3.5D MoE inference. Based on expert hotness, weight loading cost, migration overhead, and runtime resource pressure, HCRMap dynamically determines which experts should be promoted, retained, demoted, or evicted. It then maps routed token groups to suitable resident replicas, thereby jointly mitigating communication, memory, and queue bottlenecks. Experimental results show that HCRMap reduces end-to-end latency by 43.6% and 43.0% over Hydra in the prefill and decode stages, respectively; by 34.5% and 33.1% over MoEntwine; and by 46.7% and 46.0% over PIMoE.