EMO: Pretraining Mixture of Experts for Emergent Modularity

Abstract: LLMs are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge. Mixture-of-Experts (MoEs) seemingly offer a potential alternative by activating only a subset of experts per input, but in practice, restricting inference to a subset of experts for a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser. We introduce EMO, an MoE designed for modularity-the independent use and composition of expert subsets-without requiring human-defined priors. Our key idea is to encourage tokens from similar domains to rely on similar experts. Since tokens within a document often share a domain, EMO restricts them to select experts from a shared pool, while allowing different documents to use different pools. This simple constraint enables coherent expert groupings to emerge during pretraining using document boundaries alone. We pretrain a 1B-active, 14B-total EMO on 1T tokens. As a full model, it matches standard MoE performance. Crucially, it enables selective expert use: retaining only 25% (12.5%) of experts incurs just a 1% (3%) absolute drop, whereas standard MoEs break under the same setting. We further find that expert subsets in EMO specialize at semantic levels (e.g., domains such as math or code), in contrast to the low-level syntactic specialization observed in standard MoEs. Altogether, our results demonstrate a path toward modular, memory-efficient deployment of large, sparse models and open new opportunities for composable architectures.

• The paper introduces a pretraining regime (EMO) that enforces document-level expert selection to induce emergent semantic modularity.
• The methodology groups tokens by document to allow experts to specialize in semantic areas such as math, code, and technical domains.
• Empirical results demonstrate minimal performance loss under expert pruning, enabling efficient, composable inference in large Mixture-of-Experts.

Emergent Modularity in Pretraining Mixture-of-Experts with EMO

Motivation and Problem Statement

Pretrained LLMs have advanced using large-scale dense architectures, resulting in monolithic computation and deployment. Recent scaling trends have popularized Mixture-of-Experts (MoE) architectures, where only a learned subset of experts is activated per input. These models in principle offer modularity and efficiency for domain-specific inference. However, typical MoEs do not natively support meaningful modularity: activating only a relevant subset of experts for a specific domain leads to severe performance collapse due to the lack of coherent specialization among experts. As models grow increasingly sparse and large, the inability to support selective inference limits their practical deployment, especially in memory- or latency-constrained scenarios.

EMO Architecture and Methodology

EMO introduces a pretraining regime for MoEs that explicitly encourages emergent modularity without hand-crafted domain labels or human priors. The method leverages a simple yet effective constraint: all tokens within the same document are restricted to selecting experts from a shared pool; this pool varies across documents. Since documents naturally aggregate semantically related content, this constraint provides a weak, unsupervised signal that steers expert assignment toward high-level semantic grouping. During pretraining, EMO relies solely on document boundaries to enforce this expert pool restriction.

EMO scales to a 14B total parameter configuration with a 1B active parameter MoE, trained on 1T tokens. Modular grouping emerges implicitly: experts specialize at the semantic level (e.g., math, code, technical domains), a property not observed in standard MoEs, which generally specialize on low-level syntactic features.

Empirical Results

EMO exhibits several key empirical advantages over standard MoEs:

• Modular Retention: When activating only 25% (12.5%) of the expert pool, EMO experiences a negligible 1% (3%) absolute drop in performance, preserving utility. Standard MoEs exhibit catastrophic degradation in equivalent settings.
• Domain Specialization: Experts in EMO self-organize to semantic groupings (e.g., math, code), supporting the selection and composition of functionalities. This is in contrast to standard MoEs, where expert specialization remains shallow.
• Full Model Parity: With all experts active, EMO attains parity with standard MoEs on conventional benchmarks.

These results validate EMO’s approach to achieving true modularity and composability in LLMs without manual architecture design or supervision.

Theoretical and Practical Implications

The findings demonstrate that weak, document-level grouping constraints are sufficient for emergent high-level specialization in large MoEs. This suggests that domain-specialized computation and modular deployment can be induced naturally during unsupervised pretraining, not requiring explicit annotation. For practical deployment, EMO enables sub-expert selection: only relevant experts can be loaded for inference, offering substantial memory and computational savings. The semantic modularity also opens the possibility for composable inference: different expert pools can be dynamically assembled and reused for downstream tasks.

Future Directions

The results invite further research in several directions:

• Granular Modular Control: Extending document-level grouping to finer- or coarser-grained controls (e.g., sections, topics, or user-defined domains)
• Fully Composable Architectures: Using EMO as a foundation to explore plug-and-play architectures, enabling the dynamic combination of expert capabilities at inference time
• Transfer and Continual Learning: Investigating how modular experts adapt or are reused across sequential or continual learning setups
• Cross-modal Extension: Generalizing modular MoE principles to multi-modal architectures

Conclusion

EMO presents a pretraining method for MoEs that achieves semantically meaningful emergent modularity without supervision or domain priors. The architecture supports robust, modular inference with minimal performance degradation under expert pruning, outperforming standard MoEs in selective expert usage scenarios. These properties make EMO a compelling direction for scalable and efficient large model inference, composability, and memory/resource-constrained deployment. The emergent grouping behavior highlights new avenues for modular and adaptive neural architectures in NLP and beyond.