SMEAR-MoE: Soft Merging in MoE Architectures

• SMEAR-MoE is a modular neural architecture that applies a weighted parameter merge to enable fully differentiable adaptive routing and dense gradient flow.
• It replaces non-differentiable discrete expert selection with a soft merging mechanism, boosting accuracy in vision-language and multilingual ASR tasks.
• The approach improves training stability and expert specialization, offering efficient, interpretable performance gains with minimal computational overhead.

SMEAR-MoE (Soft Merging of Experts with Adaptive Routing – Mixture-of-Experts) is a fully differentiable, modular neural architecture that addresses classical challenges in sparse expert selection and conditional computation. SMEAR-MoE achieves adaptive routing by constructing a merged expert through a weighted average of all expert parameters, providing dense gradient flow, stability, and efficiency in both vision-language and speech recognition domains. The approach eliminates the need for high-variance gradient estimation techniques and improves empirical performance and expert specialization relative to conventional discrete sparse routing. SMEAR-MoE has been validated in large-scale natural language understanding and multilingual automatic speech recognition tasks, demonstrating significant accuracy improvements and interpretability.

1. Principle of Soft Merging in MoE Architectures

Traditional Mixture-of-Experts (MoE) models enable modularity by maintaining a set of specialized expert subnetworks $\{f_i(\cdot;\theta_i)\}_{i=1}^E$ along with a router mechanism that selects the most relevant expert(s) per input. Discrete routing, such as top-$k$ expert selection, introduces non-differentiability, requiring high-variance gradient estimators (e.g., REINFORCE, Gumbel-Softmax) with inherent optimization difficulties and often adverse effects on training stability and capacity utilization.

SMEAR-MoE replaces the discrete expert selection with a parameter-level soft merge. For a given input activation $u$ and a router whose output is a probability vector $R(v)\in\Delta^E$, the merged expert parameter is computed as

$$\theta_{\mathrm{merge}} = \sum_{i=1}^E R(v)_i \theta_i,$$

and the block output is

$y = f(u; \theta_{\mathrm{merge}}).$

This strategy seamlessly integrates expert selection with parameter interpolation, rendering the entire computational pathway $$v \mapsto R(v) \mapsto \theta_{\mathrm{merge}} \mapsto y$$ fully differentiable and backpropagation-friendly (Muqeeth et al., 2023).

2. Mathematical Formulation and Training Dynamics

SMEAR-MoE leverages a lightweight adaptive router—typically a linear mapping plus softmax—which yields a fully differentiable, real-valued distribution: $$R(v) = \mathrm{softmax}(Wv + b), \quad R(v) \in \Delta^E,$$ where $v$ is a pooled latent representation.

Gradients propagate through both the merging operation and the router, e.g.,

$$\frac{\partial L}{\partial \theta_i} = R(v)_i\,\frac{\partial L}{\partial \theta_{\mathrm{merge}}},$$

guaranteeing that all experts receive nonzero updates. The router parameters $W$, $b$ are updated via the chain rule, circumventing the reliance on policy gradients common in discrete routing (Muqeeth et al., 2023).

The framework is directly compatible with standard losses (e.g., cross-entropy, CTC) and requires no additional regularization for routing. In multilingual ASR, SMEAR-MoE can incorporate load-balancing objectives,

$$L_\mathrm{load} = \lambda M \sum_{m=1}^M (\bar{g}_m)^2,$$

to encourage uniform expert usage and prevent expert collapse (Pandey et al., 27 Jan 2026).

3. Application Domains and Architectures

3.1 Vision and Language Processing

In NLU settings (e.g., T5-GLUE, ResNet-DomainNet), SMEAR-MoE employs adapter-style blocks or ResNet-based experts, where each expert is architecturally identical. The soft parameter merge enables dynamic utilization of several experts per sample. SMEAR-MoE outperforms oracle tag routing, REINFORCE, Gumbel-Softmax, parameter-matched dense adapters, and expert ensembles in both average GLUE score (81.6% ±1.0) and DomainNet accuracy (62.0% ±0.1), with wall-clock throughput matching top-1 routing (Muqeeth et al., 2023).

3.2 Multilingual Speech Recognition

In multilingual LLM-based ASR, SMEAR-MoE is instantiated as a stabilized multi-expert projector between a frozen speech encoder and LLM decoder. The projector comprises a convolutional downsampler, $M$ expert MLPs, and a gating network that produces per-token expert weights $G\in\mathbb{R}^{T'\times M}$, averaged to form mixture coefficients $\bar{g}$. The merged expert parameters are then computed, and the resultant projection is input to the LLM.

Empirical evaluations on Indic languages confirm that SMEAR-MoE achieves a relative WER reduction of 7.6% compared to a single-projector baseline, maintains near-identical runtime efficiency, and exhibits clear, interpretable expert specialization aligned with linguistic families (Pandey et al., 27 Jan 2026).

4. Computational Considerations and Scalability

The computational complexity of SMEAR-MoE is near that of a single-expert MoE. For $L$ tokens and $E$ experts with $d\to m$ adapters, SMEAR-MoE requires approximately $4dmL + 2Edm$ FLOPs, where the additional merging cost is negligible for practical values $E \ll L$. In ASR, the real-time factor increase is minimal (from 0.196 for a single projector to 0.198 for SMEAR-MoE), whereas static ensembles incur a more significant slowdown (Muqeeth et al., 2023, Pandey et al., 27 Jan 2026).

Unlike sparse top-$k$ routing, where only winning experts receive gradients, SMEAR-MoE’s merging ensures dense update paths, preventing expert collapse—especially important in low-resource and multilingual regimes.

5. Expert Specialization, Interpretability, and Sharing

Qualitative analyses show that SMEAR-MoE’s routing distributions are typically sparse and interpretable. In GLUE, per-block router distributions often recover task-style expert usage, facilitating both task-specific specialization and sharing among semantically similar tasks. In DomainNet, certain visual domains share experts while others remain distinct, reflecting latent structure.

In multilingual ASR, per-language average routing heatmaps demonstrate that SMEAR-MoE recapitulates linguistic groupings (e.g., Hindi and Marathi, both Indo-Aryan, share an expert; Tamil, a Dravidian language, focuses on a different expert), with no explicit family supervision—providing insight into model internals and potential for interpretability in cross-lingual scenarios (Muqeeth et al., 2023, Pandey et al., 27 Jan 2026).

6. Limitations and Future Directions

SMEAR-MoE is limited to scenarios with homogeneous expert architectures; heterogeneous experts require parameter-alignment techniques such as permutation matching. In settings demanding fine-grained (per-token) routing for very long sequences, the merge cost per token could reach that of full ensemble evaluation.

Open research avenues include scaling to larger expert sets, exploring advanced weight-merging (e.g., Fisher-weighted averages), integrating LoRA or IA³ adapters, further application in massively wide MoEs (as in Switch Transformers), and extension to token-level SMEAR for granular routing and code-switching in multilingual models (Muqeeth et al., 2023, Pandey et al., 27 Jan 2026).

7. Comparative Summary and Impact

The following table summarizes SMEAR-MoE’s empirical advantages relative to alternative MoE and ensemble strategies:

Model/Setting	Avg. GLUE (%)	Avg. WER (%)	Throughput
Tag routing (oracle)	78.5	—	High
Single projector (ASR)	—	30.3	0.196 RTF
Static ensemble	81.7	28.8	Low (0.243)
Discrete/Top-$k$ MoE	<78.5	30.6–29.7	High
Dense Adapter	80.2	60.8 / —	High
SMEAR-MoE	81.6	28.0	0.198 RTF

In summary, SMEAR-MoE provides a robust, scalable, and interpretable mechanism for adaptive, modular computation in a variety of expert-based architectures. Its fully differentiable soft-merge approach ensures stable training, circumvents gradient estimation pitfalls, and enables both specialization and sharing consistent with underlying task or linguistic structure (Muqeeth et al., 2023, Pandey et al., 27 Jan 2026).