KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction

Abstract: Transformer-based LLMs cache context as key-value (KV) pairs during inference. As context length grows, KV cache sizes expand, leading to substantial memory overhead and increased attention latency. This paper introduces KVzip, a query-agnostic KV cache eviction method enabling effective reuse of compressed KV caches across diverse queries. KVzip quantifies the importance of a KV pair using the underlying LLM to reconstruct original contexts from cached KV pairs, subsequently evicting pairs with lower importance. Extensive empirical evaluations demonstrate that KVzip reduces KV cache size by $3$-$4\times$ and FlashAttention decoding latency by approximately $2\times$, with negligible performance loss in question-answering, retrieval, reasoning, and code comprehension tasks. Evaluations include various models such as LLaMA3.1, Qwen2.5, and Gemma3, with context lengths reaching up to 170K tokens. KVzip significantly outperforms existing query-aware KV eviction methods, which suffer from performance degradation even at a 90% cache budget ratio under multi-query scenarios.

• The paper presents KVzip, a novel method that utilizes context reconstruction for query-agnostic KV cache compression.
• It employs chunked scoring to efficiently manage long contexts, reducing computational complexity from quadratic to linear.
• KVzip achieves up to 70% cache eviction while maintaining inference accuracy and integrating with quantization for enhanced performance.

Query-Agnostic KV Cache Compression

Introduction

The paper "KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction" (2505.23416) introduces a novel approach to compressing key-value (KV) caches used by transformer-based LLMs, specifically during the inference phase. KV caches store context information as key-value pairs, which are later utilized by the models to modulate their responses based on historical data. Although essential for maintaining models' efficiency, these caches can become extensive and unwieldy as context sizes increase, leading to significant memory overhead and increased latency in attention mechanisms.

The primary contribution of this work is the development of KVzip, a robust method for query-agnostic KV cache eviction that uses context reconstruction techniques to optimize cache reuse. Unlike traditional methods that depend on immediate query information to score or evict cache entries, KVzip evaluates the intrinsic importance of the entries. This is achieved by simulating the reconstruction of contexts from compressed KV pairs, enabling the identification of entries that contribute minimally to inference tasks.

Methodology

Importance Scoring

KVzip's core methodology revolves around the importance scoring of KV pairs. It determines which entries are pivotal for reconstructing the original context and purges those with lower importance. The guiding hypothesis is that if the cache enables an accurate reconstruction of the original context, it is comprehensive enough for effective inference across diverse queries.

The process begins with a context encoded into key-value pairs via the LLM during pre-fill operations. KVzip then quantifies the importance of these pairs using attention scores derived during a forward pass tasked with reconstructing the context. Importantly, the maximum attention each pair receives during this reconstruction is measured, and scores are assigned accordingly. This maximization across query dimensions allows KVzip to maintain practical computational complexity and resource usage.

Chunked Scoring for Scalability

The scalability of KVzip to handle long contexts—a necessity for modern LLM tasks—is addressed by chunking the context. Specifically, the method computes importance scores chunk-by-chunk, reducing quadratic complexity $O(n_c^2)$ to linear complexity $O(m n_c)$, where $m$ is the chunk size. This chunking permits efficient processing without excessive memory usage, ensuring that even the largest contexts (over 120K tokens) can be efficiently managed.

Experimental Results

The empirical evaluation of KVzip across various benchmarks demonstrates its efficacy. Compared with existing methods like SnapKV, PyramidKV, and $\text{H}_2\text{O}$, KVzip showcases superior performance, maintaining accuracy even when up to 70% of the cache is evicted. It outperforms baselines across different datasets, such as SQuAD, GSM8K, and SCBench, highlighting its robustness in handling tasks ranging from question answering to code comprehension and mathematical reasoning.

Furthermore, KVzip integrates seamlessly with other optimizations, such as KV cache quantization, achieving compression ratios as aggressive as 40% without inducing significant performance loss. This compatibility extends its applicability to quantized models as well, underscoring its versatility.

Implications and Future Work

KVzip offers substantial practical benefits by significantly reducing memory overhead and enhancing inference efficiency. It aligns well with current trends towards deploying LLMs in resource-constrained environments and opens avenues to extend real-world applications like personalized conversational agents or enterprise systems using pre-computed document caches.

Future developments might explore further integration with retrieval-augmented generation scenarios, expand context-independent eviction strategies, or refine the chunk-based scoring to enhance its adaptability to evolving model architectures.

Conclusion

KVzip represents a significant stride towards improving the efficiency of LLM deployments by introducing a query-agnostic approach to KV cache compression. This work not only mitigates the extensive memory demands associated with growing context sizes but also enhances the inference process across diverse queries. Its adaptability and proven effectiveness make it a compelling tool for the current and future landscape of AI applications.