Multi-Vector Index Compression in Any Modality

Abstract: We study efficient multi-vector retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos, but its computation and storage costs grow linearly with document length, making it costly for image-, video-, and audio-rich corpora. To address this limitation, we explore query-agnostic methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC). AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation. Evaluating these methods on retrieval tasks spanning text (BEIR), visual-document (ViDoRe), and video (MSR-VTT, MultiVENT 2.0), we show that attention-guided clustering consistently outperforms other parameterized compression methods (sequence resizing and memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index. The source code is available at: github.com/hanxiangqin/omni-col-press.

• The paper introduces Attention-Guided Clustering (AGC) as a hybrid method to efficiently compress multi-vector indices while preserving critical retrieval performance across modalities.
• It demonstrates that AGC maintains at least 97% of the uncompressed retrieval quality and can even outperform uncompressed indices on certain video tasks.
• The method’s effective token utilization and modality-agnostic design highlight its potential for scaling late-interaction retrieval in large multimodal search engines.

Multi-Vector Index Compression in Any Modality: Technical Summary and Implications

Problem Overview and Motivation

Scaling late interaction retrieval to multimodal corpora presents major computational and storage challenges due to the high redundancy and noise in visual and audio content. Existing multi-vector indexing methods—critical for domains where relevance is captured via token-level interactions—fail to efficiently exploit the inherent sparsity and redundancy in multimodal data. The computation and storage costs grow linearly with document length, which is intractable for large-scale realistic deployments (e.g., video search at petabyte scale). The paper "Multi-Vector Index Compression in Any Modality" (2602.21202) systematically investigates modality-agnostic, query-agnostic multi-vector index compression under a strict vector budget, ultimately proposing and evaluating a novel hybrid method, Attention-Guided Clustering (AGC), against established baselines.

Methodological Framework

Formalization of Multi-Vector Compression

Given a document encoder $\pi: d \to \mathbf{C} \in \mathbb{R}^{m \times h}$ constrained to a fixed $m$, the challenge is to construct document representations that maximally preserve discriminative information for late-interaction retrieval—even though $d$ can originate from text, image, video, or mixed modalities. The document mapping must be performed before queries are known, emphasizing the requirement for strong query-agnosticity.

Baseline Compression Methods

• SeqResize: Applies a sequence-downsampling MLP to project variable-length token embeddings to a fixed token budget $m$. This approach is straightforward but can lead to representation underutilization and collapse when over-compressing informative multimodal inputs.
• MemTok: Appends $m$ learnable "memory tokens" to the document sequence, encoding the context jointly. The representation consists solely of these token outputs. MemTok tends to over-smooth document representations, undercutting distinctiveness among compressed tokens.
• Hierarchical Pooling (H-Pool): Executes non-parametric, iterative agglomerative clustering using cosine similarity and Ward’s linkage, grouping and averaging tokens to reduce redundancy. While robust, it can suffer from sensitivity to noise and lacks semantic discrimination beyond geometric proximity.

  Figure 1: Overview of multi-vector index compression techniques: AGC (hybrid attention-similarity), MemTok (token-based), SeqResize (projection-based), H-Pool (heuristic-based).

Proposed Method: Attention-Guided Clustering (AGC)

AGC introduces a hybrid attention-similarity mechanism, consisting of:

1. Attention-based Centroid Selection: Augments the document with learned universal query tokens; computes per-token saliency by aggregating attention weights from these special queries, thereby identifying semantically salient points independent of downstream queries.
2. Hard Clustering: Assigns each document token to one of $m$ centroids (identified above) based on cosine similarity, forming clusters without soft assignment to preserve semantic diversity.
3. Weighted Aggregation: Aggregates each cluster by weighted mean, where attention saliency scores weigh token contributions, mitigating hard operation instability and allowing gradient flow.

This approach leverages both content-awareness (attention) and redundancy reduction (clustering), tailored for late interaction in high-redundancy multimodal data.

Empirical Evaluation

Datasets and Modalities

Experiments span four benchmarks across text (BEIR), visual documents (ViDoRe v2), vision-only video (MSR-VTT), and large-scale audiovisual video (MultiVENT 2.0). The design includes severe compression ratios (e.g., up to 99.6% in some tests) and robust comparisons to uncompressed and strong prior-art baselines.

Quantitative Results and Observations

• Retrieval Quality: AGC consistently maintains $\geq 97\%$ of baseline (uncompressed) nDCG@10 across all modalities. Notably, it occasionally outperforms the uncompressed index (e.g., +1–2\% R@1 on MSR-VTT)—a nontrivial result demonstrating that compactness can regularize overcompleteness and noise, especially in high-dimensional audiovisual inputs.

  Figure 2: AGC outperforms other compression techniques at high compression ratios, maintaining superior nDCG@10.

• Comparative Analysis: While MemTok and SeqResize perform adequately for text, their efficacy drops sharply on visual or video content, where redundancy and irrelevant tokens are prevalent. H-Pool offers robust non-parametric performance but stagnates at high compression due to lack of semantic guidance.
• Stability and Generalization: AGC-trained models generalize strongly across compression ratios, outperforming heuristic methods like H-Pool when deployed at unseen budgets.
• Sparsity and Utilization: Analysis reveals that only about 1–2\% of index tokens in an uncompressed model substantially contribute to retrieval; most tokens are underutilized, accentuating the wastefulness of storing full-length representations (see utilization heatmaps).

  Figure 3: Top—AGC achieves superior, balanced utilization across compressed tokens; Bottom—MemTok suffers from over-smoothing, H-Pool provides highest internal diversity, AGC finds a balance.

• Utilization-Performance Correlation: The evenness of token utilization, as measured by Coefficient of Variation and Gini Index over MaxSim strengths, strongly correlates (Pearson $r > 0.95$) with retrieval metrics. Thus, models that more evenly exploit their token budget deliver higher retrieval quality.

  Figure 4: Retrieval metric performance is highly correlated with evenness of token utilization (coefficient of variation and Gini).

Theoretical and Practical Implications

Theoretical Insights

• The finding that AGC can outperform uncompressed indices challenges the long-standing assumption that maximal information retention (i.e., all tokens) is optimal; instead, active compression can serve as a denoising and regularization procedure, especially pronounced in high-redundancy or noisy modalities.
• The strong correlation between utilization evenness and retrieval performance validates the notion that informative compression—and not simply vector truncation—drives practical retrieval effectiveness.

Practical Consequences

• AGC enables the deployment of late-interaction index structures for massive multimodal corpora (e.g., petabyte-scale video), overcoming previous storage and inference bottlenecks.
• The method is agnostic to modality and underlying encoder backbone, making it directly extensible to future multimodal LLMs and retrieval collections.

Future Directions

Potential extensions include dynamic, content-adaptive budget allocation, leveraging content-based utilization estimates to set per-document compression levels. Improving the interaction between universal query selection and downstream discriminative tasks—possibly integrated as a retrieval-aware pretraining objective—also promises further gains.

Conclusion

This paper systematically re-examines multi-vector index compression for late interaction, especially in the context of multimodal search. The proposed AGC method—melding attention-driven semantic awareness with redundancy-reducing clustering—establishes state-of-the-art performance, especially at extreme compression ratios, and provides both theoretical and empirical insights into the mechanics of effective compressed representation. These advances lay groundwork for practical, scalable, and discriminative multimodal search engines.