MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production

Abstract: As the foundational component of versatile AI applications, training an multimodal LLM (MLLM) relies on multimodal datasets with dynamic modality mixture proportions and sample length distributions. However, existing MLLM systems remain inefficient under dynamic workloads, due to statically coupled decisions of resource allocation and model parallelization between encoders and the LLM backbone. This paper presents MegaScale-Omni, an industrial-grade MLLM training system tailored for dynamic workload adaption and hyper-scale deployment. MegaScale-Omni is built upon the training scheme of encoder-LLM multiplexing with three key innovations: (1) Decoupled parallelism strategies with long-short sequence parallelism for encoders to process variable-length samples, and full-fledged 5D parallelism for the LLM backbone, both organized under a communication-efficient parallelization layout. (2) Unified encoder-LLM representations for flexible, extensible colocation, and a new paradigm of encoder-LLM joint pipeline with workload resilience. (3) Workload balancing techniques via decentralized grouped reordering in data loaders and adaptive resharding from encoder to LLM ranks. MegaScale-Omni is deployed as the foundation of our in-house large-scale MLLM training tasks with thousands of GPUs. Our experimental results demonstrate $1.27\times$-$7.57\times$ throughput improvement under production-grade dynamic workloads, as compared to four state-of-the-art systems.

• The paper presents a novel architecture that decouples modality encoder computations from the LLM backbone, enabling dynamic, workload-resilient training at scale.
• It employs innovative parallelism strategies, including long-short sequence parallelism for encoders and 5D parallelism for the LLM backbone, achieving up to 7.57× throughput improvements and 2.21× memory efficiency gains.
• The system uses decentralized grouped reordering and adaptive resharding to optimize resource utilization and prevent bottlenecks in production environments handling thousands of GPUs.

MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production

Motivation and Problem Scope

The development and large-scale deployment of multimodal LLMs (MLLMs) have driven demand for distributed training systems capable of coping with highly dynamic workloads arising due to varying modality ratios and extreme sequence length variability. Canonical parallelization architectures applied to unimodal LLMs (e.g., treating all modality encoders as lightweight embedding layers prepended to a monolithic pipeline) are severely impaired by inefficiencies and outright failures at scale when subject to realistic, non-stationary, multimodal workflows. Static resource allocation and coupled parallelism decisions lead to bottlenecked pipeline stages, memory exhaustion, suboptimal throughput, and lost GPU hours.

Figure 1: Dynamic workloads in hyper-scale MLLM training and resulting throughput degradation in existing systems.

The paper "MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production" (2605.08962) addresses this gap by introducing a new training system tailored for dynamic, industrial-grade production, with robust workload balancing, flexible resource orchestration, and demonstrated scalability to thousands of GPUs.

Architectural Overview

MegaScale-Omni builds on the central design pattern of encoder-LLM multiplexing. The key abstraction is the decoupling of modality encoder computations from LLM backbone execution, both in terms of parallelization strategy and physical placement, while maintaining efficient communication and balanced scheduling across the entire cluster.

Figure 2: System architecture of MegaScale-Omni.

Major architectural innovations:

• Decoupled parallelism: Modality encoders are parallelized using high-concurrency strategies optimized for variable-length samples (long-short sequence parallelism), while the LLM backbone employs full-fledged 5D parallelism (DP/TP/PP/SP/EP), preventing interference and communication congestion.
• Unified encoder-LLM representation: Encoders are not statically embedded within LLM pipeline code but injected via explicit Anchor abstractions, which enable non-intrusive, extensible colocation and fine-grained, uniform pipeline orchestration even under shifting workloads.
• Industrial workload balancing: Decentralized grouped reordering in data loaders enables scalable balancing of microbatch lengths and modalities without global communication, complemented by zero-redundancy filtering and adaptive resharding for efficient inter-stage and inter-rank data movement.

  Figure 3: Multimodal LLM architecture and data processing.

Parallelism and Orchestration

Long-Short Sequence Parallelism for Encoders

Dynamic data distributions (e.g., skewed sampling lengths across image, audio, text) result in highly uneven computation and runtime load among encoders. MegaScale-Omni introduces Long-Short Sequence Parallelism (LSSP):

• Short samples are processed under DP (with optional ZeRO2/3 parameter sharding).
• Long samples are distributed via Ulysses-style SP, which enables scalable sharding along the sequence axis with memory efficiency.
• Nodes shift their internal state per batch, partitioning work adaptively at runtime to minimize stragglers and device idling.

  Figure 4: Long-short sequence parallelism for encoders. Each node maintains an instance of parallelized encoders.

This dynamic DP/SP switching avoids the pathological imbalances of prior designs that enforce a static partitioning of data or rely on cost-prohibitive global reordering.

Encoder-LLM Parallelization and Data-Flow

The LLM backbone operates under highly optimized 5D parallelism, leveraging intra-node high-bandwidth links for SP and TP, and avoiding pipeline stalls by careful co-assignment of encoders and LLM stages. Unified representations insert encoders uniformly and on-demand throughout the pipeline schedule, maintaining balanced microbatch flow even as mixture ratios and sequence lengths shift.

Figure 5: The layout of encoder-LLM parallelization.

Workload Balancing and Data Handling

Decentralized Grouped Reordering

Dynamic mixing of modalities and domains aggravates sampling skew and causes persistent device underutilization. The system tackles this with:

• Grouped, decentralized reordering: GPUs are organized in locality-aware groups that exchange metadata, apply intra-group Karp balancing, and perform all-to-all redistribution of samples.
• Zero-redundancy filtering: Redundant remote data loading and broadcast are minimized, with loader state managed for consistency and resumability under complex offloading and checkpointing regimes.

  Figure 6: Decentralized grouped reordering workflow. 8 GPUs with the group size of 4, the microbatch size is 4.

Adaptive Resharding and Communication

Resharding multimodal embeddings between encoder and LLM ranks is addressed by adaptive sample sharding and symmetric data dispatch:

• For Ulysses SP, uniform sequence sharding maps well to compute distribution and allows efficient all-to-all transfer.
• Under CP, long samples are split and handled via hybrid DP+CP, exploiting all-reduce to avoid the communication asymmetries that induce stragglers.

  Figure 7: Balancing encoder-LLM resharding for triple-modality. Encoder and LLM ranks are colocated on 2 GPUs.

Experimental Analysis

Baseline Comparison and Scaling

MegaScale-Omni is evaluated against four production-grade frameworks (Megatron-LM, Megatron-Dist, AutoParallel, Optimus) on multi-hundred GPU clusters and hyper-scale deployments, with variable mixture ratios and sequence lengths up to 512K.

• Throughput: Demonstrates $1.27\times$–$7.57\times$ end-to-end throughput improvement over baselines, with improvements strongly amplified in dynamic, encoder-heavy regimes and long-context settings.

  Figure 8: Training throughput (number of processed tokens per second) across Workload-A/B/C/D and image-text mixture ratios, with fixed sequence length of 16K/8K. Missing bars indicate OOM issues occur in the specified configurations.

• Memory efficiency: Achieves up to $2.21\times$ lower peak memory utilization at bottleneck pipeline stages.
• MFU: Stable scaling, with up to 23% higher MFU under dynamic mixtures; robust even across ratio and sequence length shifts.

  Figure 9: MFU of Workload-B and 128 GPUs: (a) across mixture ratios with sequence length of 16K, and (b) across sequence lengths with image-text mixture ratio of $7:3$.

Ablation and Sensitivity

Elimination of encoder-LLM multiplexing, workload balancing, or LSSP individually induces severe throughput degradation—$50$-$60\%$ drops observed—demonstrating that each optimization is essential for maintaining high utilization in the targeted regime.

Figure 10: Performance breakdown on Workload-A/B with image-text ratio of $7:3$ and sequence length of 16K.

The system design is robust to variations in underlying parallelism configuration; improvements persist even when pipelining, offloading, or optimizer strategies are perturbed.

Practical Operation at Hyper-Scale

MegaScale-Omni is validated on production environments scaling to thousands of GPUs. Key observations include:

• Consistently high average throughput (8M tokens/s) and robust recovery from both hardware and software faults.
• Engineering techniques for MFU maximization, alleviation of data loader and checkpointing stragglers, and avoidance of communication pathologies (e.g., P2P hangs) all proven by-field operation.
• Systematic resource and work profiling exposed further avenues for optimizing network interference and pipeline bubbles.

  Figure 11: Performance of hyper-scale MLLM training in production environments with thousands of GPUs.

Implications and Future Directions

MegaScale-Omni achieves workload-resilient, flexible, and production-grade efficiency for the dynamic training of MLLMs. Its abstraction of encoder-LLM multiplexing, unified colocation representations, and decentralized workload balancing defines a generic paradigm for future high-scale model training systems.

On a practical level, this enables stable, cost-effective training for MLLMs incorporating new modalities or domain skews and underpins industrial efforts to push context lengths, modality coverage, and model scale further.

Theoretically, these results highlight the necessity of explicit scheduling and resource orchestration layers as model architectures and workflows become ever more heterogeneous.

Future work could extend these abstractions to automatic parallelism tuning, even more granular dynamic pipeline adaptation, and integration with next-generation model architectures (e.g., mixture-of-expert MLLMs, highly sparse encoders, and domain-specialized execution backends).

Conclusion

MegaScale-Omni advances the state of the art in MLLM training system design, addressing a broad range of engineering and algorithmic challenges induced by multimodal, non-stationary, long-context workloads at unprecedented scale. Its demonstrated efficiency gains—up to $7.57\times$ throughput improvements over existing systems—validate the encoder-LLM multiplexing paradigm as a foundational approach for future AI model training frameworks (2605.08962).