DTop-p MoE: Dynamic Mixture-of-Experts

• DTop-p MoE is a dynamic sparsity-adaptive Mixture-of-Experts architecture that replaces fixed Top-K routing with a cumulative probability-based Top-p criterion.
• It employs a learned gating network and layer-wise routing normalization with a PI controller to control compute and maintain stability across model layers.
• Empirical results show improved accuracy and reduced computational overhead on language and vision benchmarks compared to traditional Top-K and static Top-p methods.

Dynamic Top-p Mixture-of-Experts (DTop-p MoE) is a sparsity-adaptive Mixture-of-Experts architecture that replaces fixed Top-K routing with a cumulative probability-based “Top-p” criterion, dynamically allocating compute based on input or token difficulty. DTop-p MoE models select a minimal set of experts for each token such that their summed gate confidence reaches a chosen probability threshold, and recent advances introduce global or controlled adaptation of this threshold to precisely match computational budgets or target sparsity. These methods outperform both traditional Top-K and fixed-threshold Top-p baselines on language and vision benchmarks, supporting more efficient and effective large-scale model pre-training.

1. Architectural Foundations and Routing Mechanisms

In DTop-p MoE architectures, a standard Transformer backbone is augmented by replacing each feed-forward sub-block with an MoE layer comprised of $N$ independent experts, typically instantiated as small FFNs. Each input token representation $x \in \mathbb{R}^d$ is routed through these experts via a learned gating network, which is a single linear layer with learned weight $W \in \mathbb{R}^{N \times d}$ followed by softmax activation: $$z = W x^\top \in \mathbb{R}^N, \quad P = \operatorname{softmax}(z)$$ Here $P_i$ denotes the gate-assigned confidence that expert $i$ is best suited to process $x$.

The central innovation in DTop-p routing is the replacement of fixed-$K$ selection with a cumulative-probability rule: experts are sorted by $P_i$ in descending order, and the minimal prefix $S = \{I_1, \ldots, I_t\}$ is chosen such that $\sum_{j=1}^t P_{I_j} \geq p$, where $p \in (0,1)$ is the Top-p threshold. Experts not in $S$ receive zero weight: $$g_i(x) = \begin{cases} P_i, & \text{if } i \in S \ 0, & \text{otherwise} \end{cases}$$ This enables the system to adaptively dispatch more experts for inputs with ambiguous/flat gate distributions and fewer for confident/peaky cases (Huang et al., 2024, Jin et al., 16 Dec 2025).

An alternative normalization frequently used is to reweight the selected $P_i$ so they sum to one within $S$, as in: $$r_i(x) = \begin{cases} P_i / \sum_{j=1}^{k_p} P_{\sigma(j)}, & i \in S \ 0, & \text{else} \end{cases}$$ where $\sigma$ is the permutation sorting $P_i$ in decreasing order, and $k_p$ is the cutoff index satisfying the Top-p criterion (Jin et al., 16 Dec 2025).

2. Sparsity Control: Fixed Thresholds vs. Adaptive Targeted Routing

A key challenge in Top-p routing is controlling the overall computational cost, since the number of activated experts is variable and depends on the (learned) gate distribution and the chosen threshold $p$. Early DTop-p MoE methods fixed $p$ (e.g., $p = 0.4$), resulting in an average of $\leq 2$ experts per token and 0.7% higher accuracy than Top-2 on key QA tasks, with 12% lower active parameter count (Huang et al., 2024).

However, a static $p$ may cause drift in sparsity as the gating distribution changes during training, leading to instability or loss of compute control at scale (Jin et al., 16 Dec 2025). To address this, advanced DTop-p approaches introduce a feedback loop using a Proportional-Integral (PI) controller: after each batch, the running average number of activated experts $a_t$ is measured and compared to a user-selected target $T$ (e.g., $T = 8$). The PI controller updates $p$ to drive $a_t \to T$, ensuring predictable FLOPs and simplifying deployment. This approach robustly enforces the compute budget and outperforms both Top-k and static Top-p across scale and modalities (Jin et al., 16 Dec 2025).

3. Layer-wise Routing Normalization and Specialization

Fixed global gating thresholds do not account for shifts in gate entropy or logit scale across model layers. Lower Transformer layers tend to have flatter gate distributions (higher-entropy), while deeper layers become sharper (lower-entropy), resulting in a sub-optimal or non-uniform sparsity pattern under a global $p$ (Jin et al., 16 Dec 2025).

Dynamic Routing Normalization addresses this by rescaling and normalizing logit distributions on a per-layer basis prior to applying softmax gating: $$z^{(\ell)} = W^{(\ell)} x \ P^{(\ell)}(x) = \operatorname{softmax}\left( \theta_\ell \frac{z^{(\ell)} - \mu^{(\ell)}}{\sigma^{(\ell)}} \right)$$ where $\mu^{(\ell)}$, $\sigma^{(\ell)}$ are the mean and standard deviation of $z^{(\ell)}$, and $\theta_\ell$ is a learnable scale. Layer-normalized gates allow heterogeneous sparsity—lower layers activate fewer experts for shallow pattern recognition; upper layers can select more experts for semantically complex inputs (Jin et al., 16 Dec 2025, Huang et al., 2024).

4. Regularization and Training Objectives

DTop-p MoE models incorporate both primary and auxiliary losses to induce load balance, sparsity, and specialization:

• The principal objective is usually the next-token LM loss $L_{lm}$.
• A load-balance loss penalizes skewed expert usage and is defined as $L_b = N \sum_{i=1}^N f_i Q_i$, where $f_i$ is the fraction of tokens routed to expert $i$ and $Q_i$ its average gate probability (Huang et al., 2024).
• An entropy (dynamic) loss $L_{d} = -\sum_{i=1}^N P_i \log P_i$ regularizes the gates to be sharp (low entropy), promoting sparse routing (Huang et al., 2024).
• The total loss thus becomes $L = L_{lm} + \alpha L_b + \beta L_d$, with $\alpha$ and $\beta$ controlling regularization strength (Huang et al., 2024).
• In (Jin et al., 16 Dec 2025), standard MoE auxiliary losses are retained, and $\theta_\ell$ parameters are updated via gradient descent alongside regular model parameters.

These losses are differentiable with respect to the softmax output, ensuring end-to-end learnability of gating and control over expert allocation.

5. Empirical Performance and Scaling Results

DTop-p MoE architectures are empirically validated in large LLM and Vision Transformer pre-training. On standard language benchmarks (PIQA, HellaSwag, ARC-e, CSQA, BBH), DTop-p outperforms both dense and Top-K-MoE baselines:

• For $N=16$, $p=0.4$, average experts/token $=1.76$ (vs. exactly $2$ for Top-2).
• Outperforms Top-2 by 0.7% average (absolute), with ≈12% fewer active parameters (Huang et al., 2024).
• Particularly prominent performance gains on challenging benchmarks involving advanced reasoning (e.g., BBH, +2% absolute) (Huang et al., 2024).

In large-scale settings with enforced sparsity control:

• For $T=8$, DTop-p converges to mean $≈8$ experts/token with stdev $≈1$; fixed-p Top-p yields stdev $≈4$ and is unstable (Jin et al., 16 Dec 2025).
• DTop-p achieves 0.5–1% lower perplexity than Top-K and static Top-p and +1.9 average points on 13 NLP downstream tasks (Jin et al., 16 Dec 2025).
• Scaling expert granularity, capacity, model size, and dataset size, DTop-p outperforms Top-K at every setting, and with larger gains as model and expert scale increase (Jin et al., 16 Dec 2025).

Layer-wise analysis reveals a monotonic decay in the number of active experts: lower layers can dispatch up to $4$ experts/token for complex queries, while upper layers often select only one, aligning compute to semantic complexity (Huang et al., 2024, Jin et al., 16 Dec 2025).

6. Design Insights and Deployment Guidelines

Substantive design observations include:

• Harder tasks and ambiguous tokens systematically induce higher gate entropy and activate more experts; semantically clear or unambiguous items default to one or two experts (Huang et al., 2024).
• The trade-off between accuracy and compute can be continuously controlled by adjusting $p$ (or $T$ when using a PI controller). For NLP models, inference accuracy saturates for $p \gtrsim 0.3-0.4$; at lower $p$, compute is saved at the cost of significant accuracy drop-off (Huang et al., 2024).
• A PI feedback scheme achieves precise average sparsity targets, minimal hyperparameter sensitivity, and robust compute control independent of wandering gate statistics (Jin et al., 16 Dec 2025).
• Recommended deployment practice is to set the target average experts $T$ to match hardware/latency requirements, initialize $p$ in $[0.2, 0.4]$, and allow the PI controller to calibrate. Auxiliary losses (load balancing, entropy) should be preserved for stable expert utilization.

PI gains $(K_p, K_i)$ may require modest retuning for new architectures or modalities. Empirical work to date verifies efficacy for model sizes up to $\sim 13.6$B total parameters and $300$B tokens; further validation at frontier scales remains an open area (Jin et al., 16 Dec 2025).

7. Comparisons and Historical Context

DTop-p MoE draws inspiration from earlier approaches:

• Top-k MoE methods, such as Switch Transformer, enforce fixed compute but do not adapt to input or layer complexity.
• Nucleus (Top-p) sampling in generation resembles the Top-p expert selection but does not address compute predictability or training stability.
• EvoMoE’s “Dense-to-Sparse Gate” anneals from initially dense to sparse routing, but does not implement precise sparsity control via a feedback controller or explicit cumulative gating (Nie et al., 2021).
• Recent DTop-p MoE methods—such as those of An et al. (“Harder Tasks Need More Experts: Dynamic Routing in MoE Models” (Huang et al., 2024)) and Zhang et al. (“Sparsity-Controllable Dynamic Top-p MoE for Large Foundation Model Pre-training” (Jin et al., 16 Dec 2025))—formalize and rigorously demonstrate the benefits of adaptive, feedback-controlled Top-p routing at scale.

These developments position DTop-p MoE as a foundation for further research into heterogeneous routing, adaptive expert architectures, and modality-agnostic sparsity scaling in large foundation models.