Scaling Laws for Native Multimodal Models

Abstract: Building general-purpose models that can effectively perceive the world through multimodal signals has been a long-standing goal. Current approaches involve integrating separately pre-trained components, such as connecting vision encoders to LLMs and continuing multimodal training. While such approaches exhibit remarkable sample efficiency, it remains an open question whether such late-fusion architectures are inherently superior. In this work, we revisit the architectural design of native multimodal models (NMMs)--those trained from the ground up on all modalities--and conduct an extensive scaling laws study, spanning 457 trained models with different architectures and training mixtures. Our investigation reveals no inherent advantage to late-fusion architectures over early-fusion ones, which do not rely on image encoders. On the contrary, early-fusion exhibits stronger performance at lower parameter counts, is more efficient to train, and is easier to deploy. Motivated by the strong performance of the early-fusion architectures, we show that incorporating Mixture of Experts (MoEs) allows for models that learn modality-specific weights, significantly enhancing performance.

Scaling Laws for Native Multimodal Models: An Analytical Overview

This analysis elaborates on the exploration of scaling laws pertinent to Native Multimodal Models (NMMs), emphasizing the comparative efficacy of early-fusion versus late-fusion architectural paradigms. The research further evaluates the implications of sparsity in NMMs through Mixture of Experts (MoEs) frameworks. This paper provides valuable insights for researchers aiming to optimize model configurations in multimodal contexts, particularly with respect to computational efficiency and performance trade-offs.

The objective of the study was to demystify the relative advantages of early-fusion models, which integrate multimodal inputs at the onset, compared to late-fusion models that defer integration till deeper within the architecture. The experimental framework comprised 457 model trainings across varied architectural designs and training data mixtures, enabling a robust analysis of scaling dynamics across different configurations.

Key Observations and Numerical Outcomes

The study meticulously delineates several salient findings:

• Early vs. Late Fusion Efficacies: Contrary to prevalent assumptions, the study establishes no intrinsic advantage for late-fusion over early-fusion architectures at comparable resource allocations. It is illustrated that early-fusion models tend to exhibit slightly superior performance metrics, especially at lower scales.
• Scalability Analogous to LLMs: The scaling law frameworks indicate that NMMs mirror the scaling behavior observed in LLMs, as evidenced by similar exponents in the power-law relationships describing loss versus compute. Notably, models demonstrate consistent scaling trends regardless of the incorporated data modalities or mixtures.
• Training Token Influence on Sparse MoEs: Sparse NMMs necessitate substantial training token volumes, surpassing those required for dense models due to their higher degree of sparsity. This requirement is captured in the observed scaling exponents for sparse models, demanding more tokens relative to the growth of active parameters.
• Impact of Modality on Specialization: This paper showcases that MoEs exhibit emergent specialization across different layers, particularly in early and late stages, encapsulating modality-specific processing weights dynamically.

Practical and Theoretical Implications

The implications of these findings are multi-faceted:

• Efficiency in Training and Deployment: Expeditious training and ease of deployment are underscored for early-fusion models, offering a pragmatic advantage over late-fusion counterparts, particularly where computational budget constraints are salient.
• Architectural Selection and Token Scaling: For practitioners, selecting an early-fusion design with consideration of optimal token scaling becomes pivotal in achieving compute-optimal designs, accounting for the nature of multimodal inputs.
• Cross-Modality Integration: Insights gleaned from modality-specific behavior in Moes might be leveraged to refine future iterations of multimodal frameworks, especially in adaptive learning scenarios.

Prospects for Future Research

Given the current research's constraints, further investigations could elucidate:

• Higher Resolution Impacts: As higher resolution inputs significantly expand the vision tokens in early-fusion architectures, addressing the computational and architectural implications remains an open avenue for exploration.
• Extrapolations to Larger Scales: Robust verification of these scaling paradigms on significantly larger models could further consolidate the theoretical foundations established here.
• Extended Multimodal Specialization: Extended research into MoEs, particularly in diverse multimodal data mixtures and routing strategies, is critical to unveiling potential efficiency gains and performance enhancements.

In summation, this paper furnishes a thorough quantitative and qualitative evaluation of NMMs, positioning early-fusion as a superior paradigm in certain scenarios, while recognizing the nuanced dynamics imparted by model sparsity and training regimes. Such insights prove instrumental for ongoing advancements in the field of multimodal AI systems.