Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

Source

arXiv

arxiv.org/abs/2607.12696v1

Read the source

Source

Plain-Language Brief

Researchers propose a new method called EcoSpec for optimizing the performance of Large Language Models (LLMs) using Sparse Mixture-of-Experts (MoE). This method aims to improve decoding speed by considering the cost of expert activation during draft selection. The study demonstrates that EcoSpec reduces active expert footprints and improves end-to-end decoding speed, achieving up to 1.62x speedup on three large-scale MoE models.

What the Document Says

  • The paper discusses a method called Cost-Aware Speculative Decoding (EcoSpec) for optimizing Large Language Models using Sparse Mixture-of-Experts.
  • Existing draft selection strategies in speculative decoding primarily optimize acceptance likelihood, but they can lead to expert scattering that increases expert-weight memory traffic.
  • EcoSpec incorporates predicted marginal expert activation cost into draft selection, favoring paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set.
  • The study evaluates EcoSpec on three large-scale MoE models across various benchmarks and shows consistent reductions in active expert footprints and improvements in decoding speed, achieving up to 1.62x speedup.

Key Concepts Mentioned

  • Large Language Models
  • Sparse Mixture-of-Experts
  • Speculative Decoding
  • Draft Selection
  • Expert Activation Cost
  • EcoSpec

Claims Made by the Document

  • Existing draft selection strategies can lead to expert scattering.
  • EcoSpec improves end-to-end decoding speed.
  • EcoSpec reduces active expert footprints.

Evidence Used

  • The text mentions that high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation.
  • The study shows consistent reductions in active expert footprints and improvements in decoding speed.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce \textit{expert scattering}: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference. We propose \textsc{EcoSpec}, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, \textsc{EcoSpec} favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate \textsc{EcoSpec} on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. \textsc{EcoSpec} consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to $1.62\times$ speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.