Unity by Diversity: Improved Representation Learning in Multimodal VAEs

Abstract: Variational Autoencoders for multimodal data hold promise for many tasks in data analysis, such as representation learning, conditional generation, and imputation. Current architectures either share the encoder output, decoder input, or both across modalities to learn a shared representation. Such architectures impose hard constraints on the model. In this work, we show that a better latent representation can be obtained by replacing these hard constraints with a soft constraint. We propose a new mixture-of-experts prior, softly guiding each modality's latent representation towards a shared aggregate posterior. This approach results in a superior latent representation and allows each encoding to preserve information better from its uncompressed original features. In extensive experiments on multiple benchmark datasets and two challenging real-world datasets, we show improved learned latent representations and imputation of missing data modalities compared to existing methods.

• The paper introduces MM-VAMP VAE using a mixture-of-experts prior to flexibly aggregate modality information.
• It demonstrates improved latent representations that boost conditional generation and data imputation on benchmark datasets.
• The model effectively balances shared global structures with modality-specific details, applicable to complex data including neuroscience.

Unity by Diversity: Improved Representation Learning in Multimodal VAEs

Introduction

The fusion of diverse modalities is pivotal for a nuanced understanding of complex phenomena. Multimodal Variational Autoencoders (VAEs) serve as a promising approach for synthesizing shared and modality-specific information. Traditional architectures often impose rigid constraints by sharing encoder outputs or decoder inputs, which can lead to sub-optimal latent representations.

Proposed Methodology

The paper introduces a novel approach using a mixture-of-experts prior for multimodal VAEs, named the Multimodal Variational Mixture-of-Experts (MM-VAMP) VAE. This method replaces hard constraints with soft constraints, offering a superior latent representation by guiding each modality’s latent space towards a shared aggregate posterior.

This approach hinges on a mixture-of-experts prior, which facilitates a more flexible representation by allowing each encoding to better preserve information from its uncompressed original features. The MM-VAMP VAE markedly improves performance in tasks such as conditional generation and data imputation.

Figure 1: Independent VAEs.

Experiments and Results

Benchmark Datasets

The paper utilizes various benchmark datasets including synthetic and complex real-world datasets for evaluation:

1. PolyMNIST: Demonstrates that MM-VAMP VAE provides superior latent representation and coherent conditional generation.

   Figure 2: PolyMNIST (translated, scale=75%): every column is a multimodal tuple $\bm{X}$.

2. Bimodal CelebA: Shows the efficacy of MM-VAMP VAE in handling more complex, multimodal data with improved accuracy and generation coherence.

   Figure 3: Bimodal CelebA: three samples of image-text pairs.

Neuroscience Application

The approach is also applied to a challenging neuroscience problem involving hippocampal neural activities. By treating each subject as a unique modality, MM-VAMP VAE allows a detailed analysis of neural patterns shared across subjects while capturing individual differences—highlighting the model's potential to advance understanding in neuroscience applications.

Theoretical Insights and Implications

The MM-VAMP VAE’s approach is theoretically optimal for the tasks it addresses. Its objective function leverages contrastive learning by encouraging similarity between the unimodal posterior approximations, effectively minimizing the Jensen-Shannon divergence.

This method provides a crucial balance between capturing shared global structures and preserving modality-specific details, leading to enhanced generalization capabilities across unseen multimodal configurations.

Conclusion

The introduction of MM-VAMP VAE represents a significant step forward in the development of multimodal VAEs. By utilizing a mixture-of-experts prior, the model overcomes limitations associated with previous aggregation-based approaches, providing a robust framework for improved representation learning.

Future work may explore extending these ideas to more powerful generative models, potentially broadening the applicability and efficacy of multimodal learning paradigms in AI.

Figures

• Figure 4: Latent Representation Classification.
• Figure 5: Another Latent Representation Classification.
• Figure 6: Yet Another Latent Representation Classification.

Each figure demonstrates the model's performance across different tasks and datasets, underscoring its versatility and robustness.