KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

Abstract: Test-time scaling is a powerful approach to obtain better reasoning in LLMs, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows. KV-cache quantization can help improve this, but current methods are evaluated under prefill-like settings and errors behave differently under autoregressive decoding. We show that in the latter regime, quantization errors accumulate across timesteps, driven primarily by incorrect token scales. We introduce KVarN, a calibration-free KV-cache quantizer that applies a Hadamard rotation followed by a dual-scaling variance normalization across both axes of the K and V matrices. We find that this combination fixes outlying token-scale errors and substantially reduces error accumulation over existing baselines. KVarN establishes a new state-of-theart for KV-cache quantization on generative benchmarks, including MATH500, AIME24 and HumanEval, at 2-bit precision. A vLLM implementation of the KVarN method is available at https://github.com/huawei-csl/KVarN

• The paper introduces a calibration-free KV-cache quantization method using channel-wise Hadamard rotation and dual-dimension variance normalization.
• It demonstrates state-of-the-art performance on reasoning and code synthesis benchmarks with significant improvements over previous techniques.
• The approach mitigates error accumulation in autoregressive decoding with minimal compute and memory overhead, enabling efficient long-context inference.

KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks

Introduction and Problem Setting

The expansion of long-context inference and test-time scaling in contemporary LLMs has amplified the need for efficient KV-cache management, with memory bottlenecks limiting practical sequence length and throughput. While prior work in KV-cache quantization (e.g., KIVI, TurboQuant, Kittty) has demonstrated competitive compression at low bit-widths, these solutions are primarily evaluated under static, prefill-oriented regimes, neglecting the autoregressive dynamics of real decoding. In such dynamic settings, quantization errors accumulate across decoding steps, resulting in pronounced performance degradation, especially for reasoning tasks with deep computational chains. The paper fundamentally asserts that standard quantization schemes inadequately preserve per-token scaling, leading to magnitude outlier errors that disproportionately impact downstream model quality.

Methodology

KVarN addresses the error accumulation phenomenon through a calibration-free quantization pipeline centered on two core operations:

1. Channel-wise Hadamard Rotation: By applying a Hadamard transform along the channel dimension, the technique achieves incoherence processing, inducing Gaussianity and attenuating channel space outlier effects. This operation is computationally efficient ($O(N \log N)$) and amenable to online inference scenarios.
2. Dual-Dimension Variance Normalization: After rotation, an iterative Sinkhorn-style normalization scheme is applied to both channel and token dimensions, balancing the variance across rows and columns. This process explicitly corrects per-token scaling errors—identified as the principal driver of end-to-end degradation in quantized KV-caches.

These transformations precede a round-to-nearest quantization step, and the representation retains two scaling vectors (per channel and per token) and a zero-point for accurate dequantization. KVarN's design ensures minimal extra compute: normalization overhead (<0.2% per 128-token chunk) and a negligible increase in dequantization cost (<1.4% over single-scale baselines with scale fusion).

Key Empirical Findings

Error Accumulation Analysis

KVarN departs from the conventional static evaluation by simulating the pseudo-decode regime, where fresh KV-cache outputs are quantized at each block during generation, mimicking true inference mode and capturing cumulative error effects. Rigorous decomposition of quantization error into magnitude and directional components reveals that magnitude errors—primarily from mis-scaled tokens—dominate the outlier regimes most detrimental to performance. Experimental interventions replacing only the worst 5% of quantized vectors with high-precision alternatives yield major improvements in KL divergence, despite accounting for a minority of the overall MSE—a clear illustration of the non-uniform importance of quantization errors in sequence models.

Strong Numerical Performance

On a comprehensive suite of benchmarks—MATH500, AIME24 (math reasoning), HumanEval (code synthesis), IF-Eval (instruction following), and line retrieval—KVarN consistently matches or outperforms all prior 2-4 bit quantization baselines at 2.3 average bits per element, often with near-lossless accuracy relative to FP16:

• MATH500 (Phi-4-14B): 84.8% KVarN vs. 77.0% TurboQuant and 74.4% KIVI at 2.3/4.5/2.3 bits respectively.
• AIME24 (Phi-4-14B): 61.7% KVarN vs. 60.0% PolarQuant and 57.8% KIVI.
• HumanEval (Qwen3-4B): 88.4% KVarN vs. 86.2% TurboQuant and 86.4% KIVI.
• Instruction-Following (Qwen3-4B, Strict): 80.4% KVarN vs. 80.3% KIVI and 79.2% TurboQuant.

Crucially, in tasks sensitive to error accumulation over hundreds to thousands of generated tokens, KVarN exhibits a notably lower performance drop relative to unquantized baselines, establishing its robustness for deep-reasoning and long-sequence settings.

Theoretical and Implementation Implications

KVarN’s experimental evidence supports the hypothesis that targeting quantization outliers—rather than optimizing for mean-squared error uniformly—yields substantially improved autoregressive performance. The combination of Hadamard rotation (to distribute energy and curb channel outliers) and dual-variance normalization (to counteract row/column scale drift) achieves a synergistic effect, containing magnitude errors even in the distribution tails. This sharply contrasts with prior methods—such as single-dimension scaling, codebook quantization, or channel-importance approaches—that focus on structural or directional distortions while neglecting token-wise norm preservation.

From an implementation perspective, the dual-scaling mechanism incurs minimal extra memory (due to scale/offset storage amortized over per-group quantization) and virtually no latency penalty when fusing scaling into the main kernel. The approach is orthogonal to and compatible with token-merging and cache-eviction strategies.

Broader Impact and Future Directions

KVarN’s direct mitigation of error accumulation opens the door for practical, aggressive KV-cache quantization in production LLM deployment, enabling order-of-magnitude reductions in memory footprint without significant loss in accuracy or reliability for complex reasoning and instruction-following. This unlocks higher-throughput and lower-latency inference, especially in memory-constrained environments or edge applications.

Theoretically, the decomposition of quantization error and empirical finding that outliers drive end-task degradation challenge prevailing approaches focused on average or uniform error minimization, suggesting new criteria for model-centric compression.

Future avenues include:

• Extending dual scaling and variance normalization to structurally different architectures (SSMs, models without standard KV-caches), though limitations already noted for non-transformer models,
• Investigating end-to-end joint train-time quantization-aware fine-tuning with KVarN,
• Combining KVarN with adaptive precision or hybrid eviction strategies,
• Analyzing token- or task-specific scaling schedules to further minimize error accumulation under diverse generative workloads.

Conclusion

KVarN introduces a variance-normalized, outlier-aware quantization scheme for transformer KV-caches that effectively limits error accumulation in long-context, autoregressive decoding. By explicitly targeting token scale deviations—using a blend of incoherence processing and dual-axis normalization—KVarN achieves state-of-the-art compression/accuracy tradeoffs, particularly in deep-reasoning and long-sequence benchmarks, and does so with negligible compute and memory overhead. The technique sets new baselines for practical efficient LLM deployment and suggests new priorities for future model compression research.