Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations

Abstract: Learning effective joint embedding for cross-modal data has always been a focus in the field of multimodal machine learning. We argue that during multimodal fusion, the generated multimodal embedding may be redundant, and the discriminative unimodal information may be ignored, which often interferes with accurate prediction and leads to a higher risk of overfitting. Moreover, unimodal representations also contain noisy information that negatively influences the learning of cross-modal dynamics. To this end, we introduce the multimodal information bottleneck (MIB), aiming to learn a powerful and sufficient multimodal representation that is free of redundancy and to filter out noisy information in unimodal representations. Specifically, inheriting from the general information bottleneck (IB), MIB aims to learn the minimal sufficient representation for a given task by maximizing the mutual information between the representation and the target and simultaneously constraining the mutual information between the representation and the input data. Different from general IB, our MIB regularizes both the multimodal and unimodal representations, which is a comprehensive and flexible framework that is compatible with any fusion methods. We develop three MIB variants, namely, early-fusion MIB, late-fusion MIB, and complete MIB, to focus on different perspectives of information constraints. Experimental results suggest that the proposed method reaches state-of-the-art performance on the tasks of multimodal sentiment analysis and multimodal emotion recognition across three widely used datasets. The codes are available at \url{https://github.com/TmacMai/Multimodal-Information-Bottleneck}.

• The paper presents MIB, which applies the information bottleneck principle to generate minimal sufficient unimodal and multimodal embeddings that reduce redundant and irrelevant information.
• It introduces E-MIB, L-MIB, and C-MIB variants that optimize fusion strategies by maximizing mutual information between embeddings and task labels.
• Experimental results on CMU-MOSI and CMU-MOSEI show that the comprehensive C-MIB approach outperforms state-of-the-art models in sentiment analysis and emotion recognition.

Multimodal Information Bottleneck for Learning Representations

The paper "Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations" introduces the Multimodal Information Bottleneck (MIB) framework, optimizing unimodal and multimodal embeddings for sentiment analysis and emotion recognition. MIB enhances representation utility by minimizing redundancy and noise, using mutual information maximization for task relevance. This essay summarizes the framework's architecture, experimental efficacy, and its implications for AI.

Introduction

The MIB framework addresses multimodal learning challenges in machine learning, such as redundant information in combined embeddings, unimodally noise-filled information, and the oversight of crucial cross-modal dynamics. To this end, MIB extends traditional information bottleneck (IB) methods by focusing on both unimodal and multimodal embeddings, optimizing them using mutual information principles.

Framework Overview

Key Components

1. Information Bottleneck (IB) Principle: MIB leverages the IB principle to promote minimal sufficient embedding, maintaining maximum relevance to target predictions while minimizing non-contributive information. This is achieved by maximizing mutual information between embeddings and labels and minimizing it between embeddings and input data.
2. MIB Variants:
   • Early-Fusion MIB (E-MIB): This version first combines unimodal features into a primary multimodal representation and then applies the IB principle to create an optimized combined representation.
   • Late-Fusion MIB (L-MIB): Operates by first optimizing unimodal embeddings individually using the IB principle before fusion into a multimodal representation.
   • Complete MIB (C-MIB): A comprehensive approach combining E-MIB and L-MIB, applying IB constraints on both unimodal and multimodal levels for maximum efficiency.

Implementation Details

Architecture

• Unimodal and Multimodal Fusion Networks: These networks handle respective modality encoding and later fusion. They are flexible and can incorporate various fusion mechanisms, ensuring compatibility with different multimodal datasets.

Optimization

• Mutual Information Computation: Using variational approximations, this computation ensures that representations hold the necessary task-related information while minimizing irrelevant data, optimizing the model through a balance parameter $\beta$.
• Regularization via Reparameterization: Gaussian assumptions for distributions allow derivative computations essential for training deep networks efficiently.

  Figure 1: The Diagram of C-MIB. DNN denotes deep neural network. The fusion mechanism in our framework is flexible.

Experimental Evaluation

Benchmark Performance

Experimental results across datasets like CMU-MOSI and CMU-MOSEI demonstrate MIB's prowess. The framework consistently surpasses state-of-the-art models in metrics like accuracy, F1 score, and MAE across both sentiment analysis and emotion recognition tasks.

Fusion Strategy Evaluation

Evaluations reveal that C-MIB, with its comprehensive inclusion of IB on all fusion stages, generally outperforms isolated strategies like E-MIB or L-MIB when fusion sophistication increases. Tensor fusion demonstrated superior results across configurations due to high expressive potential, showing MIB's flexibility in leveraging advanced representation techniques.

Implications and Future Work

MIB's integration of mutual information principles into multimodal learning marks a significant stride in AI's pursuit of efficient, noise-resistant cross-modal representations. Future research may focus on enhancing the scalability of MIB, particularly in real-time applications, and exploring its adaptability to more complex and novel datasets.

Conclusion

The Multimodal Information Bottleneck framework is pivotal in evolving multimodal learning paradigms, focusing on utility-maximized, minimal noise embeddings. Its flexibility and performance viability offer fertile ground for further explorations in sentiment analysis, emotion recognition, and beyond.