DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

Abstract: Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling neural networks while maintaining computational efficiency. However, standard MoE implementations rely on two rigid design assumptions: (1) fixed Top-K routing where exactly K experts are activated per token, and (2) uniform expert allocation across all layers. This paper introduces DynaMoE, a novel MoE framework that relaxes both constraints through dynamic token-level expert activation and layer-wise adaptive capacity allocation. DynaMoE introduces a principled routing mechanism where the number of active experts per token varies based on input complexity. Concurrently, the framework implements six distinct scheduling strategies for distributing expert capacity across network depth, including descending, ascending, pyramid, and wave patterns. We theoretically analyze the expressivity gains of dynamic routing and derive bounds on computational efficiency. Through extensive experiments on MNIST, Fashion-MNIST, CIFAR-10 (image classification), and Recycling-the-Web (language modeling) across multiple model scales, we demonstrate that DynaMoE achieves superior parameter efficiency compared to static baselines. Our key finding is that optimal expert schedules are task- and scale-dependent: descending schedules (concentrating capacity in early layers) outperform uniform baselines on image classification. For language modeling, optimal schedules vary by model size, descending for Tiny, ascending for Small, and uniform for Medium. Furthermore, dynamic routing reduces gradient variance during training, leading to improved convergence stability. DynaMoE establishes a new framework for adaptive computation in neural networks, providing principled guidance for MoE architecture design.

• The paper introduces a dynamic token-level expert activation mechanism that adjusts the number of active experts based on input complexity.
• It proposes novel layer-wise scheduling strategies (descending, ascending, pyramid, and others) to optimize expert allocation across network layers.
• Empirical evaluations show improved convergence, parameter efficiency, and accuracy compared to traditional fixed Top-K routing approaches.

DynaMoE: Dynamic Routing and Layer-Wise Adaptive Expert Allocation in Mixture-of-Experts Networks

Introduction

DynaMoE introduces a flexible, theoretically-grounded Mixture-of-Experts (MoE) framework that challenges two foundational assumptions of standard MoE: fixed Top-K routing (constant number of active experts per token) and uniform expert allocation (equal expert count per layer). The framework enables adaptive expert activation per token, governed by input complexity, and layer-specific expert scheduling via a suite of predefined strategies (descending, ascending, pyramid, wave, and uniform). These innovations are theoretically analyzed and empirically validated across diverse tasks and scales, revealing marked improvements in parameter efficiency, convergence, and performance relative to static baselines.

DynaMoE Routing and Layer-Wise Scheduling

DynaMoE uses percentile-thresholded gating to dynamically determine the number of active experts per token within each layer, with bounds $K_{\min} = 1$ and $K_{\max} = \lceil (1-\tau)N \rceil$. This allows computation to match token complexity in a differentiable, tractable manner. Six layer-wise expert scheduling strategies are formalized:

• Descending: Maximum expert count in early layers, decreasing monotonically to the minimum at depth.
• Ascending: Minimum expert count in the input layers, rising toward the output.
• Pyramid: Peak expert count at central layers, minimal at boundaries.
• Wave: Non-monotonic, oscillating expert allocations.

  Figure 1: Comparison of expert scheduling strategies showing expert capacity distribution across 12 layers for descending, ascending, pyramid, and uniform schedules.

This schema allows DynaMoE to tailor computational resources to the diversity and complexity profile of layer representations.

Architecture Overview

Each DynaMoE layer comprises a gating network with temperature scaling and Gaussian exploration noise. Expert activation is governed by the selected schedule and percentile-threshold routing. Residual connections and layer normalization are used throughout.

Figure 2: DynaMoE architecture for a descending schedule, allocating more experts in early layers and reducing capacity toward depth.

Theoretical Analysis

Expressivity

Dynamic token-level routing strictly increases the diversity of routing patterns compared to fixed Top-K, expanding the repertoire of piecewise-linear functions the network can implement. The combinatorial diversity grows rapidly with $K_{\max}$, fundamentally enhancing expressivity for fixed parameter budgets.

Computational Efficiency

Expected per-token cost is $O(d^2 \cdot \mathbb{E}[K(\mathbf{x})])$, where $\mathbb{E}[K(\mathbf{x})]$ is tunable via the threshold $\tau$. DynaMoE allows parameter counts to be decoupled from forward computation, maintaining sparse execution.

Gradient Variance and Training Stability

Dynamic routing increases routing entropy, yielding more balanced expert utilization and reducing gradient variance (bounded as $$\text{Var}(\mathbf{g}_{\text{dyn}}) \leq \text{Var}(\mathbf{g}_{\text{fixed}})(1-\gamma/N)$$ for excess entropy $\gamma$). This improves convergence stability empirically.

Schedule Optimality

The optimal expert schedule is task and diversity-profile dependent. Early-layer diversity and loss curvature drive optimality for descending schedules in spatial/hierarchical tasks; depth-wise increasing diversity favors ascending schedules in sequential/contextual domains.

Empirical Results

Image Classification

Across MNIST, Fashion-MNIST, and CIFAR-10, descending schedules consistently outperform uniform and MLP baselines, with up to $5.47\%$ accuracy improvement on CIFAR-10. Uniform schedules yield intermediate gains, and ascending schedules underperform. Descending schedules also achieve the fastest convergence and highest parameter efficiency.

Figure 3: Expert activation heatmaps showing distinct layer-wise activation profiles for descending, uniform, and ascending schedules.

Scaling analysis shows the performance gap widens as models grow larger, indicating that optimal resource allocation becomes increasingly important in high-capacity regimes.

Figure 4: Performance comparison for language modeling tasks with best validation perplexity and accuracy, emphasizing superiority of descending schedules across scale.

Language Modeling

On the Recycling-the-Web dataset, optimal expert schedules are scale-dependent: descending for Tiny models (best DynaMoE PPL $1011.80$ vs.\ uniform's $1078.31$, $6.2\%$ gain), ascending for Small (best DynaMoE PPL $2308.29$, outperforming MLP $2311.02$), uniform for Medium ($2383.89$, $3.4\%$ improvement over MLP $2468.16$). Validation accuracy differences are minor, but perplexity is sensitive to schedule.

Figure 5: Training dynamics for next token prediction: descending schedules converge fastest for Tiny, ascending/pyramid for larger models.

Figure 6: Performance heatmap for best validation perplexity across model sizes and configurations, illustrating task-dependent schedule optimality.

This empirically confirms that schedule selection must be attuned to representational diversity profiles rather than adopted universally.

Training Stability and Expert Utilization

Dynamic routing shows clear gains in expert utilization entropy and convergence behavior. Descending schedules concentrate activation in early layers; ascending schedules shift activation toward depth. This matches task structure: in vision, early layer diversity is maximal, while in language modeling, deeper abstraction is required.

Figure 7: Training loss convergence comparing schedules; descending achieves fastest initial convergence for small models, ascending for larger ones.

Analytical Discussion

Extensive theoretical and empirical analysis is provided for schedule-task interaction:

• Representational Entropy Collapse: Early layers require high expert density due to maximal input diversity.
• Loss Landscape Curvature: Piecewise-linear approximation is most beneficial where curvature is highest, i.e., early layers.
• Kolmogorov Complexity: Complexity decreases with depth in hierarchical tasks, reinforcing descending schedules.
• Gradient Propagation: Descending schedule maximizes independent gradient pathways at critical input junctures.
• Ensemble Diversity: High early-layer expert count mitigates co-adaptation and improves specialization.

For language modeling, ascending/pyramid schedules better match the increasing diversity from syntactic/semantic integration in deeper layers.

Practical and Theoretical Implications

DynaMoE advances adaptive computation principles in neural architecture design. Task-specific schedules leverage representational diversity profiles for optimal expert allocation, challenging the de facto uniform allocation standard. The approach is applicable to both MLP and Transformer-based systems, with attention-MoE coupling and superposition pressure proxies suggested for future refinement.

Practical guidelines for small-to-medium models: descending schedules (vision), ascending/pyramid (language modeling); $N_{\max}=8$, $N_{\min}=1$, $\tau=0.7$, $T=0.5$.

Limitations

Key limitations include the restricted scale of language modeling experiments, omission of standard Switch Transformer and expert-choice MoE baselines, compute fairness caveats, and reliance on hand-designed schedules. Large-scale Transformer evaluation, attention-based diversity measurement, and learned scheduling strategies are necessary as future directions.

Conclusion

DynaMoE demonstrates that both dynamic token-level expert activation and layer-wise adaptive capacity allocation yield strong, theoretically and empirically justified improvements over static MoE and dense baselines. Optimal expert schedules are task- and scale-dependent, governed by representational diversity profiles. These findings establish a principled foundation for future, adaptive MoE architecture design at both MLP and Transformer scales.