Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration

Abstract: The quadratic complexity of Multimodal LLMs (MLLMs) with respect to sequence length poses significant computational and memory challenges, hindering their real-world deployment. While existing training-free token reduction methods aim to address these inefficiencies, how to precisely identify redundant visual tokens and recover the essential information from the discarded tokens remain unclear. In this paper, we propose a ''filter-correlate-compress'' framework that decomposes the token reduction into three stages: filtering redundant tokens, correlating discarded information to preserved tokens, and compressing tokens to minimize redundancy. Following the framework, we propose a solution FiCoCo to identify limitations in single redundancy assessment, propose adaptive strategies to retain critical information from discarded tokens, and mitigate semantic dilution during token fusion. Two specialized variants, FiCoCo-V (for vision encoders) and FiCoCo-L (for LLM decoders), further optimize efficiency across MLLM architectures. Extensive experiments demonstrate that FiCoCo achieves up to 5.7x/14.7x FLOPs reduction with 92.8%/93.6% performance retention on LLaVA-1.5-7B/LLaVA-NeXT-7B. Our methods consistently outperform state-of-the-art training-free approaches, showcasing effectiveness and generalizability across model architectures, sizes, and tasks without requiring retraining. Our project page is at https://ficoco-accelerate.github.io/.

• The paper introduces a unified filter-correlate-compress paradigm to systematically reduce tokens in MLLMs, achieving up to an 82.4% FLOP reduction.
• It details a three-stage process—filter, correlate, compress—that retains key information with minimal accuracy loss.
• Empirical results across 10 benchmarks demonstrate that the method outperforms state-of-the-art training-free techniques.

Insights into "Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for Training-Free Acceleration"

The paper introduces a novel approach for accelerating Multimodal LLMs (MLLMs) by rethinking and redefining the strategy for token reduction. Recognizing the inefficiencies inherent in current training-free token reduction methodologies, the authors propose a structured paradigm aimed at enhancing inference performance without the requisite of retraining the models. The primary contribution of this research is the development of a "filter-correlate-compress" paradigm, which decomposes the process of token reduction into three distinct stages. This paper is significant in its attempt to unify varying token reduction methods under a common framework, providing clarity and rationale to methodologies that were previously treated as disparate.

Paradigm Overview

The central focus of the research is the "filter-correlate-compress" paradigm. This paradigm delineates a systematic approach to token reduction:

1. Filter Stage: This initial stage is concerned with determining which tokens are candidates for discarding. By calculating a redundancy score for each token, this stage identifies tokens that can potentially be removed without substantial loss of information.
2. Correlate Stage: Once redundant tokens are identified, this stage evaluates how the discarded tokens' information can be retained in the remaining tokens. This involves computing a correlation matrix that tracks the relationship between discarded and preserved tokens.
3. Compress Stage: The final stage involves the update of remaining tokens by fusing information from discarded tokens. This stage uses the correlation matrix to ensure that the compressed set of tokens retains as much information as possible from the original set.

Empirical Validation

A suite of methods grounded in the proposed paradigm, collectively referred to as FiCoCo, showcases significant promise. These methods—specifically FiCoCo-V (reducing tokens in the visual encoder), FiCoCo-L (reducing tokens in the LLM decoder), and FiCoCo-VL (reducing tokens in both phases)—are meticulously crafted to implement targeted token reduction strategies.

The experimental results are revealing. Across 10 multimodal benchmarks, the FiCoCo methods not only achieve up to an 82.4% reduction in floating point operations (FLOPs), but do so with only a minimal impact on accuracy, often surpassing state-of-the-art training-free methodologies. Notably, the introduction of intricate metrics such as the redundancy and correlation scores provides a flexible foundation for future methodologies, allowing for both preservation of crucial information and an acceleration of computations.

Methodological Insights

The paper's exploration of existing methods unearthed issues such as excessive coupling and lack of clarity. By providing a unified framework, it addresses these challenges and presents a paradigm where each of the stages is clearly delineated and can be independently optimized. This modularity allows researchers to iterate and enhance specific components without affecting the whole process, potentially spurring further innovations in MLLM acceleration.

Implications and Future Directions

The implications of this study are profound, particularly in contexts where efficient MLLM deployment is critical. By significantly reducing computation requirements while maintaining performance, the paradigm opens new possibilities for deploying advanced models on resource-constrained devices.

Furthermore, the authors’ approach highlights how theoretical advancements can lead to practical improvements in model performance. Future research might extend the paradigm to incorporate even more dimensions of token information or adapt the methodology for other emerging models beyond MLLMs. The ability to integrate and test new theoretical components within a unified framework could drive advances in AI model optimization more broadly.

In conclusion, this paper is pivotal for researchers focusing on MLLM deployment efficiencies, clarifying the unclear landscape of token reduction, and offering a structured method for future explorations in model acceleration.