Contaminated Mixture of Experts

• Contaminated Mixture of Experts are models integrating expert subcomponents where adversarial attacks, outlier noise, or mismatched data impact performance.
• Robust estimation incorporates modified EM algorithms, softmax gating, and robust regression techniques to mitigate contamination effects.
• These models are critical in federated learning, semi-supervised contexts, and domain adaptation, enabling resilient performance under real-world disruptions.

A contaminated mixture of experts (MoE) refers to a class of MoE models or training setups in which certain expert components—or their data, updates, or latent assignment variables—are subject to contamination. Contamination may take the form of adversarial attacks, outlier or heavy-tailed noise, model poisoning, mismatched expert structure, or noisy gating assignments. Such setups arise naturally in robust statistics, federated learning, semi-supervised learning, domain adaptation, and adversarial robust machine learning. The literature on contaminated MoE spans robust aggregation in federated settings, robust regression and clustering, adversarial vulnerabilities in neural MoE, parameter identification theory, and semi-supervised expert learning.

1. Formal Definitions and Model Variants

In contaminated MoE, the data-generation process, expert densities, or mixing proportions are explicitly designed—or implicitly assumed—to involve some fraction of samples or experts that do not conform to the nominal model hypothesis. Common formalizations include:

• Component-contaminated MoE: Each expert’s output density is a contamination mixture, such as

$$\mathcal{CN}(y\mid\mu,\sigma^2,\alpha,\eta) = (1-\alpha)\,\mathcal{N}(y;\mu,\sigma^2) + \alpha\,\mathcal{N}(y;\mu,\eta\,\sigma^2),$$

where $\alpha$ is the contamination proportion and $\eta>1$ is a scale inflation for outliers (Mambondimumwe et al., 18 Jan 2026, Mirfarah et al., 2020).

• Adversarial expert contamination: In distributed or federated setups, a subset of model updates or local experts may be adversarially corrupted, stale, or poisoned, requiring robust aggregation (Parsaeefard et al., 2021).
• Heterogeneous/frozen expert contamination: In transfer learning or prompt-based fine-tuning, a pre-trained frozen expert is blended with a trainable adapter, and the model must contend with possible overlapping or mismatched knowledge (“contamination”) (Yan et al., 31 Jan 2026, Yan et al., 24 May 2025, Yan et al., 2024).
• Noisy gating/assignment contamination: In semi-supervised learning, the latent assignment between clusters in unlabeled data and experts in the supervised task is contaminated by a noisy transition matrix (Kwon et al., 2024).
• Backdoor attacks via expert contamination: In neural MoE architectures, dormant experts can be purposefully infected and promoted to dominate output via routing triggers (Wang et al., 24 Apr 2025).

These variants are unified by the presence of an explicit or implicit contamination mechanism affecting mixture components, assignments, or expert architectures.

2. Theoretical Analysis: Identifiability and Estimation Rates

A foundational concern in contaminated MoE models is identifiability and minimax-optimal parameter estimation rates under contamination. Several regimes have been delineated:

• Distinguishability and expert merging: If a trainable expert is structurally indistinguishable from a pre-trained expert (e.g., both Gaussian with the same regression mean), the mixture becomes non-identifiable and estimation rates deteriorate. Formally, if the only solution to a linear combination

$$\eta_0 f_0 + \eta_1 f(\cdot|\eta_1) + \eta_2 f(\cdot|\eta_2) = 0$$

(almost surely) is the trivial one, the models are distinguishable; otherwise, rates slow due to merging or algebraic interactions (Yan et al., 2024, Yan et al., 24 May 2025).

• Minimax rates in regression and classification MoEs: In regimes where experts are heterogeneous and distinguishable, maximum likelihood achieves $\widetilde O(n^{-1/2})$ parametric rates for all parameters. Where contamination or merging occurs, the rates degrade, sometimes as slowly as $n^{-1/4}$ or worse, depending on the prompt-expert proximity or PDE coupling effects (Yan et al., 31 Jan 2026, Yan et al., 2024, Yan et al., 24 May 2025).
• Lower bounds: Matching minimax lower bounds are established via local Hellinger (or KL) metric arguments and Le Cam or Fano constructions, confirming that no estimator can surpass these rates up to logarithmic factors.

This analysis guides architectural and regularization choices: expert heterogeneity is essential for sample-efficient estimation.

3. Model Estimation: Algorithms and Implementation

Contaminated MoE estimation primarily extends the standard EM framework, with additional latent variables or customized procedures to model contamination:

• EM for contaminated experts: The E-step introduces, besides the component assignment, an additional latent indicator for contamination status within expert components, giving rise to two-level imputation (Mambondimumwe et al., 18 Jan 2026, Mirfarah et al., 2020).
• Robust weighting and softmax gating: In adversarially contaminated federated MoE, aggregation is performed via convex optimization to minimize deviation from a trusted server model, or with softmax gating to exponentially down-weight outlier experts (Parsaeefard et al., 2021).
• Semi-supervised and noisy assignment: Hybrid procedures estimate X-side clustering via a Gaussian mixture, and learn a noisy $\Pi_{Z|\tilde Z}$ assignment transition and robust expert regressions by least trimmed squares (Kwon et al., 2024).
• Neural backdoor attacks: In neural MoE, gradient-based trigger search and expert-freezing mechanisms are implemented to contaminate routing and infect dormant experts (Wang et al., 24 Apr 2025).

Convergence monitoring (e.g., log-likelihood increment), model selection (e.g., BIC/ICL), and initialization protocols (e.g., K-means, weighted least squares) are adapted to cope with additional contamination variables or non-standard expert structures.

4. Robustness, Outlier Handling, and Practical Effects

Contaminated MoE models are designed to resist the deleterious effects of heavy-tailed observations, adversarial updates, or outlier assignments:

• Outlier detection and clustering: Models such as contaminated Gaussian MoE (CG-MoE) and its semi-parametric variant can simultaneously cluster and flag outliers by explicit computation of contamination posteriors $v_{ik}$ (Mambondimumwe et al., 18 Jan 2026).
• Heavy-tailed error modeling: Using $t$-distribution experts (TMoE) or scale mixture models enables MoE to downweight extreme residuals, conferring high robustness relative to standard Gaussian MoE (Chamroukhi, 2016).
• Robust prediction under adversaries: MoE-FL and similar robust aggregation architectures maintain high predictive accuracy even under mass poisoning (e.g., 50–75% attackers), where standard aggregation collapses (Parsaeefard et al., 2021).
• Noisy assignment trimming: In semi-supervised MoE, least trimmed squares provides robust recovery of expert regressors up to $\sim 40\%$ label mismatch contamination (Kwon et al., 2024).

Empirically, these designs yield substantially lower parameter MSE and better held-out log-likelihood and prediction accuracy when contamination or poisoning is present, with robust performance degrading gracefully to nominal MoE in clean data.

5. Adversarial Threats and Security Implications

Recent work has exposed vulnerabilities and attack surfaces unique to contaminated and neural MoEs:

• Backdoor attacks in expert routing: By poisoning dormant experts and crafting input triggers that manipulate the gating network, an attacker can cause targeted misprediction while remaining largely undetectable by usage, feature-space clustering, or standard input filtering (Wang et al., 24 Apr 2025).
• Theoretical constructs (dominating expert): Sparse or low-usage experts can, under parameter manipulation, become dominating in the overall mixture prediction, enabling highly effective model hijacking.
• Defense limitations: Experiments show that common defenses—e.g., input filtering, fine-tuning, or clustering—are inadequate against backdoored MoEs, especially when attack triggers are semantically or statistically fluent.

This line of research highlights the need for specialized MoE-aware defenses, randomization in routing, or architectural regularization.

6. Design Recommendations and Future Directions

Core insights and recommendations for building and deploying contaminated MoEs include:

• Enforce expert heterogeneity: Structural divergence between trainable and frozen experts ensures identifiability and optimal convergence rates (Yan et al., 31 Jan 2026, Yan et al., 24 May 2025, Yan et al., 2024).
• Robust initialization and monitoring: Methods such as K-means assignment, penalized EM, and cross-validation for kernel bandwidths or contamination parameters facilitate stable inference under contamination (Mambondimumwe et al., 18 Jan 2026, Mirfarah et al., 2020).
• Trusted small public data for aggregation: In federated robust MoE, the quality and purity of the server-side trusted data are critical for gating-based defense against poisoning; dynamic bootstrapping can partially mitigate trust requirements (Parsaeefard et al., 2021).
• Adaptive gating and decentralized extensions: Extensions include adaptive gating functions (beyond distance, e.g., data-driven), fully decentralized federated topologies, and privacy-preserving or cryptographic trusted data mechanisms.
• Securing neural MoE architectures: Addressing adversarial dominance and dormant expert threats requires architectural innovations, certified randomness in expert selection, or dynamic runtime defenses.

Theoretical and empirical work continues to advance contaminated MoE, with ongoing focus on robustness, efficient learning under contamination, defense against sophisticated attacks, and generalization to complex, high-dimensional, or heterogeneous domains.