Native Multimodal Models (NMMs)

• Native Multimodal Models are unified machine learning architectures that process tokens from diverse modalities end-to-end without relying on modular components.
• They employ advanced tokenization, early fusion, and mixture-of-experts techniques to efficiently merge and reason across vision, language, and other data types.
• Practical applications span vision-language tasks, 3D generation, and real-time embodied AI, underpinned by empirical scaling laws and cutting-edge benchmarks.

Native Multimodal Models (NMMs) are a class of machine learning architectures trained end-to-end to jointly perceive, process, and generate across multiple modalities—most commonly vision and language—without compositional, adapter-based, or late fusion components. Their emergence marks a shift from modular, pipeline-based vision–language systems to unified architectures capable of native cross-modal reasoning and generation, scaling coherently as general-purpose world models.

1. Definitions and Architectural Principles

Native Multimodal Models are defined by unified backbone architectures in which the core model (often a Transformer) directly ingests and manipulates tokens from multiple modalities without the need for separately pre-trained encoders, adapter modules, or late-fusion heads. All modalities are represented as tokens or embeddings within a shared space and are processed by a single sequence modeling stack. NMMs differ fundamentally from “compositional” or “modular” multimodal models, which rely on discrete unimodal backbones (e.g., a vision encoder and a text decoder) joined later via cross-modal adapters, cross-attention, or gating layers (Cui et al., 30 Oct 2025, Diao et al., 16 Oct 2025, Shukor et al., 10 Apr 2025).

Key architectural variants include:

• Early-fusion architectures: All modalities are tokenized and merged at the input layer; all Transformer blocks jointly mix all modalities, often from the first layer onward (Shukor et al., 10 Apr 2025, Diao et al., 16 Oct 2025).
• Dense unified backbones: No modality-specific branches at depth; both vision and language tokens (and potentially audio, 3D, etc.) propagate through the same self- and cross-attention layers (Cui et al., 30 Oct 2025, Li et al., 2024).
• Sparse mixture-of-experts (MoE): Recent NMMs introduce MoE layers that activate modality-specialized expert weights, enabling both efficient scaling and retention of modality-specific capacity within one backbone (Li et al., 2024, Shukor et al., 10 Apr 2025, Tian et al., 9 Oct 2025).

NMMs are distinguished by:

• Shared next-token objectives spanning all modalities
• End-to-end optimization of all parameters from scratch or strong LLM initialization
• Unified tokenization and positional schemes to resolve spatial/sequential alignment

2. Core Methodologies and Mechanisms

Tokenization and Fusion

All modalities are discretized or embedded into a common $d$-dimensional token space; e.g., images via vector-quantized VAEs or patchification (Cui et al., 30 Oct 2025, Chen et al., 17 Oct 2025), text via learned tokenizers (Li et al., 2024, Cui et al., 30 Oct 2025), and 3D or video via structured latents or tokenized grids (Ye et al., 2 Jun 2025, Xie et al., 18 Jun 2025). Tokens are often interleaved in the input and passed through the same positional and embedding layers, commonly with generalized Rotary Positional Embeddings (RoPE) augmented for both 1D (temporal) and 2D/3D (spatial/volumetric) alignment (Diao et al., 16 Oct 2025, Li et al., 2024).

Unified Modeling Objectives

NMMs are trained primarily via autoregressive next-token prediction across multimodal sequences: $$\mathcal{L} = - \sum_{t=1}^{T} \log p(x_t \mid x_{<t})$$ with $x_t$ drawn from the union of all modality-specific vocabularies (Cui et al., 30 Oct 2025, Chen et al., 17 Oct 2025, Li et al., 2024). Joint objectives with flow-matching or diffusion-based losses are adopted for image/video generation (Chen et al., 17 Oct 2025, Xie et al., 18 Jun 2025, He et al., 30 Dec 2025).

Specialized Conditioning and Cross-Modal Attention

Unified models may integrate “soft prompts” or context signals between modalities (e.g., AR hidden states conditioning diffusion denoisers (Chen et al., 17 Oct 2025)), or apply cross-attention/cross-modal query mechanisms inline, ensuring compositional reasoning while enabling high-fidelity rendering.

Reinforcement Learning (RL) and Post-Training

NMMs increasingly exploit RL for fine-tuning, using reward models targeting prompt alignment, multi-object composition, OCR/text rendering, or domain-specific criteria, e.g. via Group Relative Policy Optimization (GRPO) (Chen et al., 17 Oct 2025, Cui et al., 30 Oct 2025). Post-training on curated or instruction data further boosts alignment and consistency, especially for editing and multi-turn interaction (Chen et al., 17 Oct 2025).

3. Scaling Laws, Efficiency, and Practical Design

Empirical Scaling Laws

Large-scale studies indicate that under equal compute, early-fusion NMMs are as effective or superior to late-fusion models: $$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$ with optimal parameter–data trade-offs favoring early-fusion and MoE extensions at moderate size ($N\propto C^{0.53}$ for dense, $N\propto C^{0.36}$ for MoE; $D\propto C^{0.66}$ for MoE) (Shukor et al., 10 Apr 2025, Tian et al., 9 Oct 2025). Mixture-of-Experts further enhances scaling and supports specialization with minimal increase in compute cost (Li et al., 2024, Shukor et al., 10 Apr 2025).

Model Efficiency and Deployment

NMMs obviate the need for dual backbones or vision adapters, reducing inference and training overhead. MoE architectures in models like Aria activate only a sparse subset of parameters per token (∼8 experts out of 66), enabling high throughput and dynamic specialization (Li et al., 2024). Scaling to long contexts (e.g., 64k tokens) and high-resolution modalities is straightforward—rotary frequency bases and patchification schemes scale accordingly (Li et al., 2024, Cui et al., 30 Oct 2025).

Inference optimizations, notably Discrete Diffusion Adaptation (DiDA), convert naive autoregressive sampling into parallelizable discrete denoising, yielding >20× acceleration with parity in generation quality (Cui et al., 30 Oct 2025).

4. Applications and Empirical Benchmarks

Vision–Language Tasks

NMMs achieve state-of-the-art or near state-of-the-art performance on wide-ranging multimodal tasks:

Model	MMMU val	DocVQA	MMLU 5-shot	HumanEval	GenEval multi-object	ImgEdit (GPT-4)
Aria	54.9	92.6	73.3	73.2	—	—
BLIP3o-NEXT	—	—	—	—	0.91	3.62
Emu3.5	—	—	—	—	—	4.41

NMMs like Emu3.5, BLIP3o-NEXT, and Aria match or surpass leading open-source and some proprietary models in T2I, X2I, image editing, long-context reasoning, and video understanding benchmarks (Cui et al., 30 Oct 2025, Chen et al., 17 Oct 2025, Li et al., 2024).

3D and Beyond-2D Modalities

NMMs generalize to 3D object generation and understanding (e.g., ShapeLLM-Omni, N3D-VLM), incorporating voxelized or lifted representations and grounding directly in 3D spatial logic (Ye et al., 2 Jun 2025, Wang et al., 18 Dec 2025). Video-capable models such as Show-o2 unify images and videos in a joint causal VAE latent space, applying autoregressive and flow-based techniques natively (Xie et al., 18 Jun 2025).

Multilingual and Culturally-Native Retrieval

Native approaches to multilingual VL tasks (e.g., training exclusively on captions written by native speakers) show measurable performance gains over translation-based models, highlighting the value of “native” perceptual grounding (Buettner et al., 2024).

Multimodal Knowledge Graph Completion

NativE demonstrates the capacity for handling diverse and imbalanced real-world knowledge graphs by adaptively fusing and adversarially augmenting modalities (structure, text, image, audio, video, numeric) natively within a unified scoring framework, leading on all benchmarks (Zhang et al., 2024).

Embodied and Real-Time Omnimodal AI

NMMs such as RoboEgo instantiate native full-duplexity across vision, audio, text, and action—fusing all modalities at each transformer step and providing human-level latencies and conversational responsiveness, a milestone for embodied and agentic AI (Yao et al., 2 Jun 2025).

5. Open Challenges, Limitations, and Future Directions

• Extensible Modality Coverage: While state-of-the-art NMMs natively fuse text, images, audio, video, and even 3D, integrating less-structured data types (point clouds, tactile, olfactory, sensorimotor streams) remains an open research direction (Yao et al., 2 Jun 2025, Wang et al., 18 Dec 2025).
• Scaling to Data Constraints: Scaling laws for small/medium-sized NMMs indicate positive returns when allocating parameters proportionally to both language and vision branches, but empirical saturation points and optimal scaling rules for non-vision modalities require further study (Tian et al., 9 Oct 2025, Diao et al., 16 Oct 2025).
• Data Quality, Cultural Salience, and Naturalness: High diversity and native-culture data are critical for ceiling performance, especially in multilingual or open-world tasks (Buettner et al., 2024). Caption augmentation and joint pre-training on genuinely native data remain vital.
• Interpretability and Reasoning Transparency: NMMs with explicit geometric or visual reasoning (e.g., N3D-VLM, DiffThinker) show improved interpretability, but most unified architectures remain black boxes (Wang et al., 18 Dec 2025, He et al., 30 Dec 2025).
• Safety, Alignment, and Adversarial Robustness: Native integration simplifies pipeline security but raises challenges for fine-grained safety alignment, content filtration, and adversarial robustness (Chern et al., 2024, Cui et al., 30 Oct 2025).
• Standardization and Ecosystem Development: Modular “native primitive blocks” (e.g., Pre-Buffer in NEO) and open evaluation pipelines are being promoted to facilitate democratized research and extensibility (Diao et al., 16 Oct 2025, Li et al., 2024).

6. Paradigm Implications and Ecosystem Trends

Native Multimodal Models move the field toward truly unified, modality-agnostic next-token prediction engines, facilitating seamless long-context, long-horizon, and high-fidelity cross-modal reasoning, synthesis, and world modeling:

• Unified Next-Token Generators: Models such as Emu3.5, Aria, and NEO treat images, text, and potentially other modalities as equivalent tokens, enabling long-horizon reasoning, interleaved generation, and autonomous exploration (Cui et al., 30 Oct 2025, Li et al., 2024, Diao et al., 16 Oct 2025).
• Reusable Native Primitives: Drop-in native primitives and modular blocks (MHNA, Native-RoPE, multi-branch MoE) allow rapid conversion of language or vision LLMs into unified NMMs, supporting broad ecosystem expansion (Diao et al., 16 Oct 2025, Li et al., 2024).
• Foundation for Embodied and Interactive AI: Full-duplexity, streaming, and real-time capabilities now demonstrated by RoboEgo, Emu3.5, and similar systems suggest a trajectory toward general-purpose, embodied native agents (Yao et al., 2 Jun 2025, Cui et al., 30 Oct 2025).
• Theory-Grounded Design: Recent work formalizes scaling laws and efficiency properties, providing concrete guidelines for architecture and data budget allocation in future NMM construction (Shukor et al., 10 Apr 2025, Tian et al., 9 Oct 2025).

Native Multimodal Models, as an architectural and methodological paradigm, hence provide a scalable, interpretable, and robust foundation for multimodal AI, with demonstrated performance across generation, understanding, world modeling, and embodied reasoning tasks. Continued investigation into scaling, extensibility, interpretability, and safety is expected to shape the next phases of NMM research and deployment.