Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

Source

arXiv

arxiv.org/abs/2607.28418v1

Read the source

Source

Plain-Language Brief

Researchers introduce WIDE, a dynamic token-level pruning framework for improving the efficiency of large language models (LLMs). The framework allows each token to select attention and FFN channel groups dynamically, providing finer-grained computation allocation than existing methods. This leads to better quality retention and performance boosts when compared to state-of-the-art dynamic depth pruning under certain settings.

What the Document Says

  • The text discusses a new token-level dynamic width pruning framework called WIDE for improving the efficiency of large language models (LLMs).
  • WIDE enables fine-grained computation allocation by allowing each token to dynamically select attention-head groups and FFN-channel groups.
  • Through a two-stage training pipeline, WIDE learns effective token-wise sparse execution patterns.
  • The researchers claim that WIDE achieves substantially better quality retention than existing approaches.

Key Concepts Mentioned

  • large language models (LLMs)
  • token-level dynamic width pruning
  • attention-head groups
  • FFN-channel groups
  • two-stage training pipeline

Claims Made by the Document

  • WIDE provides a performance boost of 55.1% when compared to the state-of-the-art dynamic depth pruning under calibration-only settings.
  • Under prefill and decoding inference workloads, WIDE achieves close-to-theoretical kernel-level speedups of up to 1.98x for prefill and 4.95x for decoding.

Evidence Used

  • At 50% sparsity, WIDE provides 55.1% performance boost...
  • Under prefill and decoding inference workloads, WIDE achieves close-to-theoretical kernel-level speedups of up to 1.98x for prefill and 4.95x for decoding...

Methods / Approach

  • token-level dynamic width pruning
  • two-stage training pipeline

Limitations Stated by the Authors

The authors mention that existing static structured pruning methods often cause substantial accuracy degradation under aggressive sparsity.

Limitations Noted by the Agent

Agent interpretation — not from the source.

It's not explicitly stated if there are any limitations due to the agent's understanding or capabilities.

Open Questions

  • What is the impact of WIDE on a variety of LLM models and tasks?
  • How does WIDE perform under different levels of sparsity?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text focuses on the benefits of WIDE, but it doesn't provide information about potential drawbacks or limitations.

Raw Abstract / Excerpt

Pruning is a promising approach for improving the efficiency of LLMs. Existing static structured pruning methods are hardware-friendly and can deliver practical throughput gains, but their input-agnostic computation allocation often causes substantial accuracy degradation under aggressive sparsity. Recent dynamic sparsity methods improve quality retention by adapting computation to individual inputs, yet they remain largely limited to coarse-grained structural decisions and their practical acceleration under real-world inference scenarios remains challenging. To address these challenges, we present WIDE, the first end-to-end differentiable token-level dynamic width pruning framework designed for both prefill and decode scenarios. WIDE enables fine-grained computation allocation by allowing each token to dynamically select attention-head groups and FFN-channel groups, extending dynamic pruning beyond layer-level decisions to neuron-block-level granularity. Through a two-stage training pipeline, WIDE learns effective token-wise sparse execution patterns and achieves substantially better quality retention than existing approaches. To make such fine-grained dynamic pruning practical, we further propose a pruning--kernel co-design framework that decomposes dynamic sparsity acceleration into mask reordering, hardware-agnostic block-level skipping, and hardware-dependent intra-block skipping, enabling efficient execution across different granularities. At 50% sparsity, WIDE provides 55.1% performance boost when compared to the state-of-the-art dynamic depth pruning under calibration-only settings. Under prefill and decoding inference workloads, WIDE achieves close-to-theoretical kernel-level speedups of up to 1.98x for prefill and 4.95x for decoding, as well as 1.68x and 1.55x end-to-end acceleration. Our code is available at https://github.com/EIT-NLP/LLM-Pruning/tree/main/WIDE.