CompKV: Compensation-Aware KV Selection for Long-Context LLM Inference

Source facts

CompKV selects token blocks jointly with the downstream compensation mechanism for sparse attention rather than choosing tokens only by attention mass. The preprint derives a block-level residual approximation based on attention mass and within-block logit variation and reports speedups over full attention on RULER and LongBench-Pro.

Analysis

The design makes cache selection and compensation a coupled optimization problem. That is relevant to long-context serving because memory traffic, not only arithmetic, is a primary bottleneck as context lengths grow.

Uncertainty/Doubt

• This is an arXiv preprint; speedup, quality preservation, and baseline comparisons were not independently reproduced here.