Rosetta: Composable Native Multimodal Pretraining

Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge. However, accommodating continuous generative objectives alongside discrete understanding tasks causes severe gradient conflicts. Existing architectures, including standard Mixture-of-Experts (MoE), are highly susceptible to representation overwriting. Even structurally partitioned paradigms like Mixture-of-Transformers (MoT) remain vulnerable to catastrophic forgetting, severely impeding multimodal scalability. In this work, we introduce Rosetta, a composable native multimodal pretraining framework designed for seamless and non-destructive modality expansion. Rosetta adopts a modular paradigm where core foundational knowledge is preserved within global shared experts, while modality-specific capabilities are distributed across plug-and-play experts. To guarantee non-destructive composition, we propose Momentum-Anchored Orthogonal Projection (MAOP). MAOP leverages the optimizer's momentum state as an implicit semantic anchor, selectively neutralizing conflicting gradient components from new modalities while preserving synergistic updates. Extensive evaluations demonstrate that, while standard MoE and MoT architectures suffer catastrophic forgetting of previously acquired knowledge, Rosetta robustly preserves established language and visual understanding. Furthermore, it delivers superior image generation and unlocks cross-modal synergy, paving the way for truly composable and unified multimodal foundation models. To facilitate further multimodal research, we release our code and checkpoints to the community. Project page at https://rosetta-lmm.github.io/.

• The paper introduces a novel framework that integrates language and visual modalities without catastrophic forgetting by leveraging the MAOP mechanism for gradient orthogonalization.
• It employs unified attention and composable FFN routing with a global shared expert to maintain cross-modal alignment and support non-destructive expansion.
• Empirical results demonstrate Rosetta’s superiority over MoE and MoT, achieving stable language metrics and high-fidelity visual generation.

Composable Native Multimodal Pretraining via Rosetta: Architecture, Optimization, and Empirical Analysis

Introduction

Rosetta presents a composable native multimodal pretraining framework targeting the non-destructive integration of heterogeneous modalities—discrete (language understanding) and continuous (visual generation)—in large-scale foundation models. The core motivation stems from the limitations of contemporary sparse architectures, such as standard Mixture-of-Experts (MoE) and structurally partitioned Mixture-of-Transformers (MoT), which are highly vulnerable to representation overwriting and catastrophic forgetting during functional expansion. Rosetta is constructed to overcome the fundamental Forgetting-Synergy Dilemma: efficiently expanding generative capabilities without degrading foundational knowledge, and simultaneously facilitating synergistic cross-modal learning.

Rosetta proposes both architectural and optimization-level innovations. Architecturally, it partitions expertise into global shared experts (the semantic anchor) and plug-and-play, modality-specific experts. On the optimization front, the novel Momentum-Anchored Orthogonal Projection (MAOP) mechanism neutralizes conflicting cross-modal gradient interference with zero memory overhead by leveraging the optimizer’s momentum state as a dynamic anchor. Extensive empirical results demonstrate Rosetta’s efficacy in preserving and augmenting language and visual understanding while achieving strong performance in visual generation.

Figure 1: Left—MMLU dynamics across compositional pretraining stages highlight the catastrophic collapse in MoE and MoT, while Rosetta maintains performance stability; Right—Rosetta enables high-fidelity image generation while preserving foundational knowledge.

Rosetta Architecture

Rosetta’s architectural principles are summarized as follows:

• Unified Attention: Unlike approaches that enforce modality-specific QKV projections, Rosetta maintains globally shared attention across all modalities, maximizing bandwidth for dense cross-modal contextualization and preventing representational isolation at early layers.
• Composable FFN Routing: Functional expansion is strictly confined to the feed-forward layers. Modality-specific sparse experts are organized into dedicated pools (Text, ViT, VAE), with each token type routed exclusively via its own specialized gating mechanism. This prevents high-variance generative gradients from corrupting experts dedicated to discrete modalities.
• Global Shared Expert: Every token is obligatorily processed by the global shared expert, which functions as a universal semantic bridge. This ensures shared cross-modal alignment, as gradients from all tasks (language, visual understanding, generation) are aggregated by the shared parameter space.

  Figure 2: Threefold architecture—(1) unified attention for dense alignment, (2) modality-aware composable FFN routing, (3) MAOP for conflict-free optimization.

The explicit structural decoupling in the FFN, with architectural constraints on routing, is critical in eliminating destructive capacity competition and enables non-destructive modality extension, as evidenced by expert routing distribution stability during expansion phases.

Conflict-Free Optimization via MAOP

Introduction of new generative tasks fundamentally alters the loss landscape, creating high-gradient-conflict regimes in shared parameters. Conventional gradient surgery approaches for multi-task optimization (e.g., PCGrad) are computationally and memory-infeasible at scale due to $\mathcal{O}(N)$ memory overhead from instantiating task-specific gradient buffers.

The MAOP mechanism addresses this by exploiting the optimizer's running momentum as an implicit semantic anchor—effectively representing the historical trajectory of established knowledge. When a new gradient exhibits destructive interference (negative inner product with the momentum), the component antagonistic to existing knowledge is orthogonally projected out. This guarantees that only synergistic updates affect the shared parameters, fully eradicating catastrophic forgetting without incurring additional memory cost or introducing training instability under large-scale FSDP.

Experimental Analysis

Rosetta is benchmarked against strict MoE and MoT baselines, all upcycled from the Qwen3-0.6B backbone, ensuring precise parity in parameter activation, initialization, and curriculum. The evaluation spans three composable pretraining regimes: language foundation, visual understanding, and visual generative expansion.

Key findings:

• Catastrophic Forgetting Elimination: Upon integrating visual generative objectives, MoE and MoT undergo irreversible collapse in language metrics (e.g., MMLU, BBH, MBPP), with BBH/MBPP dropping to zero due to routing collapse and gradient competition. Rosetta, in contrast, exhibits minimal degradation or even improvement (MBPP), attributing stability to MAOP’s gradient orthogonalization and composable routing topology.
• Cross-Modal Synergy: Baselines can only freeze knowledge or limit expansion to prevent forgetting, precluding further performance gains. Rosetta demonstrates monotonic improvements in visual understanding post-generative expansion, surpassing its pre-T2I checkpoint across all visual understanding benchmarks, substantiating true synergistic cross-modal parameter sharing.
• Superior Visual Generation: In all synthesis metrics (FID, CLIPScore, HPSv2) and compositional adherence (T2I-CompBench), Rosetta dominates, producing high-fidelity and prompt-aligned images while avoiding semantic drift and geometry collapse observed in other architectures.

  Figure 3: Qualitative comparisons show Rosetta's superior prompt adherence and fidelity, contrasted with semantic drift and artifacts in MoE/MoT generations.

  

Figure 4: Top row—Rosetta surpasses MoE/MoT in maintaining upward MMBench trajectory, optimization bound, and capacity rate. Bottom row—Rosetta accelerates T2I convergence and stabilizes loss in visual and language domains.

Ablation studies confirm that removing the global shared expert or MAOP either severely impedes compositional generation and reasoning (without shared expert) or reintroduces catastrophic forgetting (without MAOP), thus establishing both as indispensable.

Figure 5: Routing heatmaps evidence that only Rosetta maintains stable expert selection before/after generative expansion; MoE and MoT suffer severe routing drift and interference.

Figure 6: Qualitative generations—Rosetta preserves scene geometry and object semantics, whereas MoE suffers from texture corruption and MoT omits important compositional elements.

Implications and Future Directions

Rosetta introduces a scalable paradigm for continual modality expansion and non-destructive multimodal pretraining. The MAOP optimization principle, leveraging implicit momentum instead of explicit buffers, provides a foundation for scaling conflict-aware updates in other parameter-sharing settings beyond multimodal integration. Architecturally, plug-and-play extensibility via composable routing, combined with a global semantic anchor, supports integration of arbitrarily heterogeneous objectives (including audio, temporal, and 3D modalities) without architectural retraining.

As foundation model complexity grows, ensuring stability and synergy across expanding functional domains will become increasingly central. The demonstrated monotonic scaling properties with respect to expert pool size (without sacrificing established knowledge or incurring computational overhead) indicate Rosetta's practical relevance for future AGI systems.

Conclusion

Rosetta resolves the inherent tension between stability and plasticity in composable multimodal pretraining by unifying structural and optimization-level mechanisms. The combination of modality-aware FFN routing, a global semantic bridge, and memory-free gradient orthogonalization (MAOP) achieves non-destructive, synergistic expansion of both discrete and continuous modalities within a single architecture.

This framework establishes new state-of-the-art resilience to catastrophic forgetting and cross-modal interference, enabling the tractable integration of future modalities crucial for general-purpose artificial intelligence.

Reference: "Rosetta: Composable Native Multimodal Pretraining" (2607.00293)