Multinomial Logistic Experts in MoE Models

• Multinomial logistic experts are components in MoE models for multi-class classification, using logistic regression for both gating and expert predictions.
• They enable complex nonlinear decision boundaries and interpretable model decompositions, vital for domain adaptation and transfer learning across various data types.
• Advanced EM algorithms facilitate parameter estimation in these models, ensuring optimal convergence rates even in functional and high-dimensional settings.

A multinomial logistic expert is a specific type of component within mixture-of-experts (MoE) models for multi-class classification, wherein both the gating function and the experts themselves are parameterized by (possibly functional) multinomial logistic regression forms. This paradigm generalizes the classical MoE framework by enabling complex nonlinear decision boundaries and interpretable model decompositions, and it is foundational to scalable architectures for structured prediction, domain adaptation, and transfer learning in settings ranging from standard tabular data to high-dimensional functional and neural representations.

1. Model Architecture and Formulation

A multinomial logistic expert forms the backbone of a MoE designed for multi-class (categorical) outcomes. Let $x \in \mathbb{R}^d$ be an input vector (or, in functional models, a function-valued covariate), and $y \in \{1, \dots, K\}$ a discrete class label. The MoE model specifies the conditional class distribution as a mixture:

$$P(y = s \mid x) = \sum_{i=1}^k \pi_i(x) \, f_i(y = s \mid x)$$

where each gating function $\pi_i(x)$ is non-negative and normalized ($\sum_i \pi_i(x) = 1$), and each expert $f_i$ is a multinomial logistic regression:

$$f_i(y=s \mid x) = \frac{\exp(a_{is}^\top x + b_{is})}{\sum_{\ell=1}^{K} \exp(a_{i\ell}^\top x + b_{i\ell})}$$

Classically, the gating network can take the form of a softmax ($\exp(w_i^\top x)/\sum_j \exp(w_j^\top x)$), a sigmoid aggregation, or a regularized/modified variant (Nguyen et al., 2023, Pham et al., 1 Feb 2026). In the functional data context, the expert inputs may be one-dimensional curves $X(t)$, with both the gating and expert linear predictors formulated functionally as $\int_T X(t) \alpha_{k}(t) \, dt$ or similar (Pham et al., 2022).

2. Gating Functions and Their Variants

The gating function is pivotal in partitioning the input space and modulating expert behaviors. The dominant forms include:

• Softmax Gating: Traditional multidimensional softmax, with weights $w_i$, facilitates smooth selection across experts. However, softmax gating can exhibit pathological estimation rates during parameter collapse (experts with vanishing coefficients), owing to partial differential equation (PDE) interactions between gate and expert parameters. In such scenarios, parameter estimation can degrade to sub-polynomial rates, particularly under model over-specification or expert degeneracy (Nguyen et al., 2023).
• Sigmoid Gating: Deploys elementwise sigmoids instead of softmax, with

$$\pi_i(x) = \frac{\sigma(w_i^\top x)}{\sum_j \sigma(w_j^\top x)}$$

or with parameterized amplitudes and offsets ($\exp(\gamma_i)\,\sigma(\alpha_i^\top x + \beta_i)$, etc.). Sigmoid-gated MoEs avoid the aforementioned degeneracy of softmax, maintain polynomial sample efficiency, and streamline convergence analysis in both regression and classification settings (Pham et al., 1 Feb 2026).

• Modified Gating Functions: To remedy softmax degeneracy, one can pre-transform the inputs to the gate (e.g., $x \mapsto M(x)$) to enforce linear independence between gate and expert parameter effects, thereby restoring parametric estimation rates (Nguyen et al., 2023).
• Euclidean-Score Gating: For settings requiring gating temperature parameters, replacing the inner product ($\alpha^\top x$) by an affinity function ($-\|x-\mu_i\|^2 + \beta_i$), as in various kernel or distance-based MoEs, circumvents pathological temperature-parameter interactions and yields optimal sample complexity (Pham et al., 1 Feb 2026).

3. Estimation, Optimization, and Algorithmic Properties

Maximum likelihood estimation for multinomial logistic MoEs proceeds via the expectation-maximization (EM) algorithm, with a latent expert indicator $z$ introduced per data point. The EM algorithm, for both classical and functional-input settings, involves:

• E-step: Calculation of posterior responsibilities (soft assignments to experts), leveraging current parameter estimates (Fruytier et al., 2024, Pham et al., 2022).
• M-step: Separate maximization with respect to expert and gating parameters. The expert update is a weighted multinomial logistic regression for each expert; gating parameters are updated via weighted (possibly penalized) softmax or sigmoid regression.

Recent work formally identifies EM for these models as a mirror descent procedure with the complete-data KL as Bregman divergence, yielding interpretive and convergence guarantees (Fruytier et al., 2024):

$$\theta^{t+1} = \arg\min_\theta \left\{\langle \nabla L(\theta^t), \theta - \theta^t \rangle + D_A(\theta \|\theta^t)\right\}$$

where $D_A$ is the Bregman divergence of a suitable strictly convex mirror map $A$.

Local linear convergence of EM is guaranteed when the “missing information” matrix (ratio of missing to complete Fisher information) remains bounded, typically when signal-to-noise is high, or when gating and expert parameters are well separated (Fruytier et al., 2024). Empirical and theoretical results demonstrate that EM outperforms direct gradient descent in convergence rate and final accuracy, particularly for multinomial logistic experts.

In functional settings, sparsity-promoting regularization on targeted derivatives (e.g., $L_1$-norm of second differences) enables interpretable piecewise-linear coefficient functions (Pham et al., 2022).

4. Theoretical Guarantees: Convergence Rates and Sample Complexity

The density estimation rate for multinomial logistic MoEs, parameterized by either softmax or sigmoid gates, is nearly parametric ($O_p(\sqrt{\log n/n})$) across both well-specified and over-specified regimes (Nguyen et al., 2023, Pham et al., 1 Feb 2026). However, parameter estimation rates depend critically on the interplay between gate and expert parametrization:

Gating	Regime	Exact-spec param rate	Over-spec param rate	Degeneracy risk
Softmax	No parameter collapse	$O_p((\log n/n)^{1/2})$	$O_p((\log n/n)^{1/4})$	None
Softmax	Expert parameter collapse	Subpolynomial, no polynomial	Subpolynomial, no polynomial	Yes, due to PDE linkage
Modified Softmax	All	Parametric, as above	Parametric, as above	None if transformation ensures independence
Sigmoid (modif.)	All	$O_p((\log n/n)^{1/2})$	$O_p((\log n/n)^{1/4})$	None

The introduction of temperature parameters in gating functions can, unless addressed by structural modifications (e.g., Euclidean score gating), induce exponential sample complexity due to parameter entanglement (Pham et al., 1 Feb 2026). This reveals that careful gating-function design is essential not only for expressivity but also for statistical efficiency.

5. Extensions: Functional Data and Contaminated/Transfer MoE

Functional multinomial logistic experts generalize the predictor space to curves or functions. Here, both gating and expert regression coefficients become functions (typically expanded in bases such as B-splines). The FME-EM approach promotes interpretability by penalizing the $L_1$-norm of discretized derivatives:

$$P_g(\alpha) = \chi\sum_{k=1}^K \|D^2 w_k\|_1, \quad P_e(\beta) = \lambda\sum_{k=1}^K\sum_{m=1}^G\| D^2 u_{k,m}\|_1$$

This regularization induces piecewise-linear sparsity in the estimated coefficients, allowing the discovery of intervals of predictor non-importance, and consistently yields better classification accuracy compared to unregularized or single-expert functional models (Pham et al., 2022).

In contaminated or transfer-learning MoE architectures, a pre-trained expert is integrated with a trainable adapter via a logistic gate:

$$p_G(y=s \mid x) = [1-\pi(x;\beta,\tau)] f_0(y=s \mid x,\eta_0) + \pi(x;\beta,\tau) f(y=s \mid x,\eta)$$

Here, expert heterogeneity (differing function classes for adapter and frozen expert) is crucial. Heterogeneous-expert regimes yield standard parametric estimation rates ($\widetilde O(n^{-1/2})$), while homogeneous configurations suffer slowed convergence inversely proportional to the similarity of the two experts. Thus, expert heterogeneity is theoretically preferable for sample-efficient adaptation (Yan et al., 31 Jan 2026).

6. Implications for Model Selection and Practical Design

The selection and structure of multinomial logistic experts, both in terms of gating and expert specifications, are strongly consequential for statistical consistency, convergence, and sample efficiency:

• Softmax gating may induce slow or stalled learning when component collapse occurs; preprocessing of gating inputs or employing sigmoid variants (with amplitude scaling) eliminates this risk (Nguyen et al., 2023, Pham et al., 1 Feb 2026).
• Sigmoid gating (with modified amplitudes) is robust and admits optimal rates across regimes, including over-parameterization and expert collapse settings.
• Temperature-entering-gating must be carefully controlled, using Euclidean or other non-inner-product-based affinity functions if included, to maintain polynomial sample complexity (Pham et al., 1 Feb 2026).
• Heterogeneous structure in contaminated/adapter settings guarantees minimax optimal rates and should be favored in transfer learning scenarios (Yan et al., 31 Jan 2026).
• Functional regularization (via derivative sparsity) yields interpretable coefficients and can enhance accuracy, particularly in high-dimensional or structured input domains (Pham et al., 2022).

These insights inform best practices for MoE modeling with multinomial logistic experts: prioritizing gating architectures and model parametrizations that ensure identifiability and optimal learning rates under anticipated data regimes. Empirical results across simulated, real, and transfer domains corroborate these prescriptions (Pham et al., 2022, Fruytier et al., 2024, Pham et al., 1 Feb 2026).