speculative decoding
4 notes, newest first.
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Researchers propose a new method called EcoSpec for optimizing the performance of Large Language Models (LLMs) using Sparse Mixture-of-Experts (MoE). This method aims to improve decoding speed by considering the cost…
A Practical Investigation of Training-free Relaxed Speculative Decoding
The paper investigates training-free methods to speed up the sampling process of autoregressive Language Models (LLMs) by relaxing some strict guarantees in standard speculative decoding. The authors provide insights…
A Practical Investigation of Training-free Relaxed Speculative Decoding
The study investigates training-free methods of relaxed speculative decoding, which can speed up sampling from an autoregressive Language Model (LLM) by using a faster auxiliary model to draft tokens that are then…
Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding
NVIDIA announced DFlash Speculative Decoding, an inference optimization technique for Blackwell GPUs targeting multi-agent AI workloads. The claimed performance improvement is up to 15x on NVIDIA Blackwell hardware.
