XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression

Abstract: Learning-based 3D visual geometry models have significantly advanced with the advent of large-scale transformers. Among these, StreamVGGT leverages frame-wise causal attention to deliver robust and efficient streaming 3D reconstruction. However, it suffers from unbounded growth in the Key-Value (KV) cache due to the massive influx of vision tokens from multi-image and long-video inputs, leading to increased memory consumption and inference latency as input frames accumulate. This ultimately limits its scalability for long-horizon applications. To address this gap, we propose XStreamVGGT, a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache, enabling extremely memory-efficient streaming inference. Specifically, redundant KVs generated from multi-frame inputs are initially pruned to conform to a fixed KV memory budget using an efficient token-importance identification mechanism that maintains full compatibility with high-performance attention kernels (e.g., FlashAttention). Additionally, leveraging the inherent distribution patterns of KV tensors, we apply dimension-adaptive KV quantization within the pruning pipeline to further minimize memory overhead while preserving numerical accuracy. Extensive evaluations show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$\times$ and accelerating inference by 5.48$\times$, enabling practical and scalable streaming 3D applications. The code is available at https://github.com/ywh187/XStreamVGGT/.

• The paper introduces a KV cache compression method that integrates query-guided token pruning and per-channel quantization to maintain constant memory during long-duration streaming.
• It demonstrates a 4.42× reduction in memory usage and a 5.48× speedup on benchmarks without significant loss in 3D vision accuracy.
• The approach enables scalable real-time 3D perception for AR, robotics, and SLAM, and suggests pathways for dynamic cache budgeting in future research.

XStreamVGGT: Memory-Efficient Streaming 3D Vision via KV Cache Compression

Introduction

The study introduces XStreamVGGT, a memory-efficient streaming Vision Geometry Grounded Transformer (VGGT) that addresses the exponential Key-Value (KV) cache growth inherent in prior 3D transformer-based streaming systems. Standard models such as VGGT and its streaming variant, StreamVGGT, rely on accumulating frame-wise KV pairs for online inference, but these caches grow unbounded with temporal horizon, leading to prohibitive memory and compute requirements that preclude practical long-duration deployment.

XStreamVGGT systematically integrates token pruning and advanced quantization within the streaming transformer cache pipeline, capping the memory footprint and yielding substantial throughput improvements with negligible degradation in 3D vision accuracy metrics. By leveraging inherent spatial and temporal redundancy in vision transformers, along with distribution-aware quantization schemes, the framework demonstrates order-of-magnitude efficiency gains, facilitating scalable transformer-based 3D perception.

Figure 1: The XStreamVGGT pipeline sequentially aggregates frame queries, scores KV token importance, prunes redundant historical tokens, then applies channel-/token-adaptive quantization to compress the KV cache at each streaming step.

Context and Related Work

Classical multi-view 3D reconstruction methods (e.g., SfM, MVS) involve brittle, multi-stage pipelines and lag in real-time, generalizable performance. Transformer-based systems such as DUSt3R, CUT3R, and VGGT unify the pipeline, internalize scene priors, and enable robust, feed-forward 3D inference across tasks like dense depth estimation, geometric point map regression, and camera pose estimation.

VGGT extended these paradigms with a billion-plus parameter transformer and an Alternating-Attention architecture. The streaming variant, StreamVGGT, supports video or long sequence input via frame-wise causal attention and explicit KV cache accumulation, pioneered for 3D streaming analogously to autoregressive LLMs. However, StreamVGGT inherits the classical memory blow-up from naive, unbounded KV accumulation with sequence length.

Previous KV cache compression research focuses primarily on LLMs using either importance-based KV pruning or quantization. Vision transformers, especially streaming 3D variants, feature greater redundancy due to local spacetime correlations and process frames—not individual tokens—thus their inherent geometry and cache structure demand specialized compression strategies.

XStreamVGGT: Methodology

Pruning Redundant Multi-Frame KVs

Vision tokens, owing to high intra-frame and inter-frame correlation, are frequently redundant within the KV cache. XStreamVGGT introduces a query-guided token importance scoring mechanism: current-frame queries (after head and spatial pooling) are inner product-matched against historical keys to yield importance metrics, compatible with high-throughput attention kernels like FlashAttention.

Tokens from the first and most recent frames are always retained (for geometric anchoring and current context); intermediate frames are scored and the lowest-importance tokens pruned once the cache hits a fixed length budget $\mathcal{L}_{\max}$. This converts the memory curve from linear to constant in frame count, bounding inference latency and footprint.

Figure 2: Input query frame for attention calculation in the streaming setting.

Figure 3: Visualization of channel outlier structure for Key tensors, motivating the need for channel-adaptive quantization.

Dimension-Adaptive KV Quantization

A distributional analysis (Figure 3) reveals pronounced channel-wise outliers in Key tensors (massive dynamic range disparities), whereas Value tensors are more homogeneous across channels but less so across tokens. Standard per-tensor quantization yields severe performance loss due to scale inflation from outliers.

XStreamVGGT proposes per-channel Key quantization and per-token Value quantization:

• Each Key channel uses individual quantization parameters, constraining outlier channels to benefit from finer scale granularity.
• Values are quantized at the per-token level, corresponding to their more uniform distribution.

Quantized KVs are fully compatible with hardware-efficient asymmetric quantization schemes and can utilize INT4 representations with minimal MSE quantization error inflation. All steps are tightly coupled with pruning—quantization is applied after each cache pruning instance, ensuring no memory or compute bottleneck re-emerges at any point in the sequence.

Integration and Pipeline

The overall pipeline (Figure 1) handles streaming frames by:

1. Extracting token representations.
2. Aggregating queries and scoring historical KV importance.
3. Pruning historical tokens, always keeping first and current frames.
4. Applying dimension-adaptive quantization.
5. Maintaining a constant-footprint KV cache for all subsequent streaming inference.

Experimental Results

XStreamVGGT is evaluated on multi-benchmark 3D reconstruction and depth estimation tasks, including NRGBD and 7-Scenes for 3D geometry, and TUM/ScanNet/KITTI for pose/depth. Across all tasks, the model demonstrates:

• Memory usage reduction by 4.42$\times$ and speedup by 5.48$\times$ relative to StreamVGGT, as measured on long video input benchmarks.
• Negligible degradation in 3D vision task accuracy: For NRGBD and 7-Scenes, Normal Consistency (NC) reduction is <2% compared with uncompressed baselines. Camera pose errors (ATE, RPE) increase by only 0.006–0.025 units (absolute), and monocular/video depth achieves near-parity.
• Robust behavior under extreme cache constraints (2K tokens, INT4 quantization) without mode collapse or catastrophic forgetting, validated in ablation studies.
• Qualitative results show near-indistinguishable reconstructions and depth maps compared to the uncompressed baseline.

Figure 4: Ablation study—performance with varying cache lengths evidences low redundancy requirement for streaming memory.

Figure 5: Comparisons of qualitative 3D reconstruction results between StreamVGGT and XStreamVGGT exhibit visual parity.

Figure 6: Comparative video depth estimation on diverse testframes, showing XStreamVGGT matches the baseline's perceptual quality.

Implications and Future Directions

XStreamVGGT establishes that order-of-magnitude KV cache reduction is achievable for streaming 3D transformers with minimal impact on geometric quality or camera/scene understanding. The tuning-free, distribution-adaptive quantization/pruning framework generalizes to any vision transformer requiring temporally scalable online inference and hardware-friendly implementation.

Practical deployment implications include real-time geometric perception for AR, robotics, and SLAM under fixed-resource constraints. Theoretically, the results motivate further study of redundancy structure in spacetime transformer caches and cross-domain adaptation of LLM-inspired compute-saving techniques.

Future extensions could explore dynamic cache budgeting, where pruning parameters adapt online to scene difficulty or context shift, as well as tighter encoder-decoder coupling with hardware-level quantization kernels for end-to-end streaming deployment.

Conclusion

XStreamVGGT delivers a comprehensive, tuning-free pipeline for KV cache compression in streaming vision transformers. By strategically integrating importance-based pruning and dimension-adaptive quantization, the framework caps inference memory at constant cost while preserving state-of-the-art geometric performance. This work thus advances scalable, long-horizon 3D transformer deployment, eliminating the primary resource bottleneck inherent to prior streaming architectures.

Reference: "XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression" (2602.21780)