The risk of KV cache compression

Abstract: Transformer inference on long sequences is expensive because softmax attention repeatedly reads from a large KV cache. The prevalent approach to this bottleneck is KV cache compression, which replaces the full cache with a compact summary. Despite its practical importance, the design of such summaries is largely driven by empirical experimentation. On the theoretical side, existing results show that KV cache compression can be impossible in the worst case, but offer little systematic guidance for designing algorithms in regimes where accurate compression is possible. We bridge this gap by characterizing the minimax risk of KV cache compression in terms of the intrinsic compressibility of a cache, revealing when and how accurate compression is possible. These results yield novel design principles for KV cache compression under causal masking that map efficiently to prefill and autoregressive decoding while achieving minimax-optimal risk. We instantiate these principles in a practical algorithm and report promising performance on LongBench in targeted experiments. Overall, our results provide a principled avenue for practical KV cache compression with theoretical guarantees.

• The paper presents a minimax risk formulation of KV cache compression, establishing rigorous upper and lower bounds for both query-aware and query-agnostic regimes.
• It reformulates compression as a sparse barycentric approximation in Hilbert space, using eigenvalue-tail bounds to quantify error rates.
• Efficient, parallelizable algorithms are proposed and validated to achieve nearly lossless compression with up to 95% KV memory reduction in streaming inference tasks.

Theoretical and Practical Risks of KV Cache Compression

Introduction and Problem Formulation

Transformer models, with their softmax attention-driven architectures, depend centrally on maintaining a Key-Value (KV) cache during inference. As input context length increases, the size of the KV cache grows linearly, producing computational and memory bottlenecks at inference time—particularly in long-context or streaming tasks. The main approach to alleviate this is KV cache compression: compactly summarizing the cache by replacing a large number of key-value pairs with a reduced subset or weighted summary, while maintaining accurate downstream attention outputs.

This paper rigorously formulates the KV cache compression problem as a measure approximation question: Given a probability measure $P$ representing the cache and a (potentially unknown) distribution over future attention queries $\nu$, what is the minimax compression risk—i.e., the minimum achievable mean-squared error over all compressors, measured as the maximal expected deviation in attention output across all compatible cache/query pairs, under a fixed compression budget $K$?

Theoretical Characterization: Minimax Risk of Compression

The paper provides a comprehensive characterization of the compressibility of KV caches under the lens of minimax risk. Unlike previous works that employed empirical heuristics—such as recency, attention mass, or token importance proxies—the analysis here is grounded in the intrinsic geometry of the cache as probed by softmax attention.

Compression as Sparse Approximation in Hilbert Space

The core insight is to recast KV cache compression as the problem of sparse barycentric approximation in a Hilbert space, using the response profile $\Gamma_P(k, v)$ of each key-value pair $(k, v)$, which quantifies the influence of the pair on attention outputs across possible future queries $q$ sampled from $\nu$. The response covariance operator $_{P, \nu}$, defined on the function space of queries, encapsulates the directions in Hilbert space along which the cache's contents manifest in attention computations.

Tight Upper and Lower Bounds

The central theoretical contributions include the following:

• Upper Bounds: For compressors aware of the query distribution (query-aware), the minimax risk is shown to scale as $\operatorname{tail}_{K/3}(_{P,\nu}) / K$, where $\operatorname{tail}_{r}(L)$ is the sum of eigenvalues beyond the top $\nu$0 of the covariance operator. In the query-agnostic regime, lacking access to $\nu$1, the optimal minimax risk scales as $\nu$2.
• Lower Bounds: The paper provides matching lower bounds constructs, demonstrating that under sharp attention regimes (i.e., when queries can sharply retrieve individual tokens), the upper bounds are unimprovable. Notably, the lower bounds do not require pathological, adversarial caches; extremely challenging regimes appear in natural settings even with modest redundancy.

A key implication is the sharp separation between query-aware and query-agnostic settings: having access to the (statistical) distribution of future queries enables dramatic improvements in achievable compression error rates, via adaptation to the dominant spectral structure in $\nu$3.

Graded Compressibility

The results instantiate a graded spectrum of compressibility, ranging from highly compressible caches, where most tokens are redundant for future attention, to incompressible caches embodying many independently retrievable facts. This provides not only tight quantitative error rates but also clear criteria for when and why safe compression is feasible.

Algorithmic Translation to Practical KV Compression

Building on the theoretical minimax rates, the work advances efficient, implementable parallel algorithms for cache compression in both causal prefill and autoregressive inference. By using local reducers—randomized or structure-exploiting (clustering-based)—and organizing compression as a merge-reduce tree across token blocks, the method aligns practical computation with theoretical optimality.

Design Principles

• Random Sampling Reducer: Achieves the minimax-optimal risk in the query-agnostic setting by subsampling tokens proportionally to a fixed prior over keys.
• Clustering Reducer: Projects onto leading directions of the response covariance (as approximated by a feature map) and clusters tokens in protected coordinates, yielding spectral error guarantees when future query geometry is sufficiently aligned with the chosen features.

These reducers are efficiently parallelizable, avoid extensive sequential dependencies, and support both batched prefill and streaming decoding, with a logarithmic-in-context-depth computational overhead.

Consistency with Causal Masking

The algorithms satisfy causal masking constraints, ensuring that at any position, the compressed summary only contains information from the allowable prefix, which is essential for autoregressive models.

Empirical Validation

Experiments conducted on the LongBench-v2 benchmark with Qwen3-32B validate the minimax predictions: random sampling already provides a strong compression baseline; however, using clustering along principal spectral directions enables nearly lossless compression at up to 95% reduction in the KV budget. Notably, these results are competitive with state-of-the-art practical baselines, despite the latter not providing theoretical minimax risk guarantees, and even though the proposed methods apply compression during both prefill and decoding.

Implications, Limitations, and Future Directions

The key implication is that principled, theoretically grounded KV cache compression is not only possible but also practical, provided the cache exhibits adequate spectral redundancy with respect to probable future queries. For practitioners, this analysis suggests moving beyond heuristic compression proxies and towards algorithms that explicitly account for response geometry or its surrogates.

However, the sharp query-aware/agnostic divide highlights the need for improved estimation or prediction of future query distributions—potentially via learned surrogates or online adaptation—to further close the gap in practical scenarios. The measure-theoretic formalism developed here also invites further work on cache compression under more structured models, such as hierarchical clustering, non-uniform attention dynamics, or multimodal settings.

The paper raises numerous open questions, such as generalizing to multi-head or multi-layer compression, connecting spectral structure to language/model statistics, or developing learned feature maps aligned with real downstream query patterns.

Conclusion

This work provides a principled framework for understanding and implementing KV cache compression, anchored by tight minimax risk bounds and efficient algorithms. The measure-based perspective not only explains the empirical effectiveness of cache compression but also specifies the fundamental limits and optimal algorithmic strategies for both query-aware and agnostic scenarios. This represents a step towards reliable, provable compression methods for long-context transformer inference, establishing a foundation for ongoing research into scalable, memory-efficient attention models.

Reference:

"The risk of KV cache compression" (2607.01520)