FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jesús Olivera
• Venue: arXiv: cs.AI
• Published: 2026-07-07
• Collected: 2026-07-08
• URL: http://arxiv.org/abs/2607.06519v1
• PDF: https://arxiv.org/pdf/2607.06519v1
• DOI: UNKNOWN

Plain-Language Brief

The paper introduces FreqDepthKV, an inference-time cache compression method for long-context LLM (Large Language Model) inference that reduces memory and bandwidth costs while maintaining task accuracy on various benchmarks like question answering, summarization, code generation, and needle retrieval.

What the Document Says

• FreqDepthKV is a compression method for long-context LLM inference.
• It aims to reduce memory and bandwidth costs of KV caches.
• The method factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals.
• A lightweight online probe assigns attention heads based on their contribution to reconstruction-sensitive attention logits.
• FreqDepthKV maintains task accuracy under smaller cache budgets across various benchmarks.
• It achieves a close performance to full KV while outperforming prior compressed-cache methods.
• Decoding throughput is improved to 70.4 tokens/s.
• TTFT (Time To First Response) is reduced to 2.06 seconds.
• Peak KV memory is lowered to 6.2 GB, achieving a 3.9x effective compression ratio.

Key Concepts Mentioned

• Long-context LLM
• KV cache
• Compression
• FreqDepthKV
• Inference-time
• Online probe
• Reconstruction-sensitive attention logits

Claims Made by the Document

• FreqDepthKV preserves task accuracy under smaller cache budgets.
• It outperforms prior compressed-cache methods.

Evidence Used

• Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV achieves performance close to full KV while outperforming prior compressed-cache methods.
• With a 32k-token prefill window, FreqDepthKV reaches specific scores in the text.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning. We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits, allowing the compression policy to adapt to prompt structure without retraining. Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets. With a 32k-token prefill window, FreqDepthKV reaches 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods. It also improves decoding throughput to 70.4 tokens/s, reduces TTFT to 2.06 seconds, and lowers peak KV memory to 6.2 GB, achieving a 3.9x effective compression ratio.