Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

Source

arXiv

arxiv.org/abs/2607.06523v1

Read the source

Source

Plain-Language Brief

The research introduces DepthWeave-KV, a new cache compression method for long-context language models. This method adapts to the needs of different tokens by factorizing key and value states across neighboring transformer layers using shared low-rank channel bases while retaining lightweight token-specific residuals. It also uses online error tracking during generation to adapt compression without retraining the model, resulting in improved memory usage and task quality on various benchmarks.

What the Document Says

  • The research presents DepthWeave-KV, a token-adaptive cache compression method for long-context language models.
  • DepthWeave-KV factorizes key and value states across neighboring transformer layers using shared low-rank channel bases while retaining lightweight token-specific residuals.
  • It uses a token-conditional depth router to allocate higher reconstruction rank to instruction-bearing and retrieval-critical tokens.
  • Online error tracking from attention-output probes is used for adaptation during generation without retraining the base model.
  • A fused CUDA implementation is used to jointly perform basis lookup, residual dequantization, and attention projection to reduce decode-time memory traffic.
  • Across several benchmarks, DepthWeave-KV achieves near-full-cache task quality with lower memory usage and higher retrieval accuracy compared to prior compressed caches.

Key Concepts Mentioned

  • DepthWeave-KV
  • cache compression
  • long-context language models
  • token-adaptive
  • transformer layers
  • shared low-rank channel bases
  • lightweight residuals
  • token-conditional depth router
  • online error tracking
  • attention-output probes
  • fused CUDA implementation

Claims Made by the Document

  • DepthWeave-KV improves average score and retrieval accuracy over prior compressed caches.
  • DepthWeave-KV achieves near-full-cache task quality with substantially lower memory use.

Evidence Used

  • Improvement in average score and retrieval accuracy is mentioned but no specific numbers are provided. The improvement over prior compressed caches is implied.
  • The achievement of near-full-cache task quality with lower memory use is stated, but the actual percentage or amount of improvement is not specified.

Methods / Approach

  • token-adaptive cache compression
  • shared low-rank channel bases
  • lightweight token-specific residuals
  • token-conditional depth router
  • online error tracking from attention-output probes
  • fused CUDA implementation

Limitations Stated by the Authors

No specific limitations are mentioned by the authors.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The research doesn't provide specific numbers for improvement in average score and retrieval accuracy or the exact reduction in KV memory.

Open Questions

  • What is the exact percentage of improvement in average score and retrieval accuracy achieved by DepthWeave-KV?
  • What is the exact amount of KV memory reduction achieved by DepthWeave-KV?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text mentions improvements but does not provide specific numbers. It also states the reduction in memory usage but does not specify the exact amount.

Raw Abstract / Excerpt

Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation. We introduce DepthWeave-KV, a token-adaptive cache compression method that factorizes key and value states across neighboring transformer layers using shared low-rank channel bases while retaining lightweight token-specific residuals where attention behavior is sensitive. DepthWeave-KV combines cross-depth residual factorization with a token-conditional depth router that allocates higher reconstruction rank to instruction-bearing and retrieval-critical tokens, and uses calibration-free online error tracking from attention-output probes to adapt compression during generation without retraining the base model. A fused CUDA implementation jointly performs basis lookup, residual dequantization, and attention projection to reduce decode-time memory traffic. Across LongBench, Needle-in-a-Haystack, L-Eval, and long-form QA and summarization benchmarks, DepthWeave-KV achieves near-full-cache task quality with substantially lower memory use, improving average score and retrieval accuracy over prior compressed caches while reaching 8.3x KV memory reduction and 72.8 tokens per second at 64K context.