TT-LoRA MoE: Efficient Fine-Tuning Architecture

• TT-LoRA MoE is a parameter-efficient fine-tuning approach that integrates tensor-train low-rank adapters with Mixture-of-Experts to address scalability and task interference in LLMs.
• It decouples expert training from router optimization by freezing task-specific expert adapters, thereby preventing inter-task interference and catastrophic forgetting.
• The architecture reduces parameter counts and computational costs significantly, enabling scalable multi-task learning and domain-specific adaptations.

Tensor-Train LoRA Mixture-of-Experts (TT-LoRA MoE) is a parameter-efficient fine-tuning methodology for LLMs and sequence models that unites tensor-train low-rank adapters (TT-LoRA) with the modularity and dynamic routing power of Mixture-of-Experts (MoE) architectures. This class of methods seeks to address critical scalability, continual learning, and task interference bottlenecks encountered by traditional fine-tuning and parameter-efficient approaches. TT-LoRA MoE generalizes earlier LoRA-MoE methods by substituting each low-rank LoRA adapter with a highly compressed tensorized adapter, dramatically reducing parameter counts while enabling efficient storage and dispatch of hundreds of task- or skill-specific expert modules.

1. Architectural Foundations

The canonical TT-LoRA MoE architecture is built atop a frozen LLM backbone. In each target submodule (e.g., attention projections, FFN blocks), E parallel experts are injected. Each expert module comprises a TT-LoRA adapter, i.e., a low-rank matrix update parameterized via a tensor-train decomposition. For a base weight $W_0 \in \mathbb{R}^{m \times n}$, a TT-LoRA update $\Delta W$ is constructed from a set of TT-cores $\{G^{(k)}\}_{k=1}^{p+q}$, such that $\Delta W$ can be applied via a series of tensor contractions without explicitly forming the dense matrix. The typical per-expert adapter parameter count is sublinear in $m$ and $n$ (e.g., $33{,}920$ for TT-LoRA vs $1{,}703{,}936$ for LoRA in one experiment) (Kunwar et al., 29 Apr 2025).

Each token (or, in some variations, each instance) is routed to a subset (typically one or a few) of these experts via a learned gating network. While standard LoRA requires explicit manual selection of adapters, TT-LoRA MoE automates expert selection by training a lightweight router network on frozen base representations.

2. Expert Training and Decoupled Routing

TT-LoRA MoE instantiates a strict separation between expert specialization and router training. The framework proceeds in two distinct stages (Kunwar et al., 29 Apr 2025):

• Stage 1: Independent Expert Training. For each task $i$, a dedicated TT-LoRA adapter $\phi_i$ is fine-tuned with the backbone weights $\Theta$ kept frozen. Upon convergence, each $\phi_i$ (task expert) is frozen, completely eliminating inter-expert interference and catastrophic forgetting.
• Stage 2: Sparse Router Training. A global router is trained to map pooled (or flat) base model representations $h_x \in \mathbb{R}^d$ to logits over all experts. At inference, the router dispatches each input $x$ to its selected expert, activating only one adapter per input. Only the router is trainable in this stage, while both backbone and experts remain static.

The result is a decoupled, modular system in which new experts can be added incrementally for continued learning or domain expansion, with no need to revisit or retrain the main router or previous experts.

3. Routing Mechanisms and Sparsity Control

The routing strategy in TT-LoRA MoE is typically sparse. The base architecture routes each instance to a single expert by top-1 selection over noisy logits (Kunwar et al., 29 Apr 2025). For broader MoE-LoRA variants, soft or differentiable sparse routing functions such as Sparsegen or temperature-softmax are adopted, sometimes paired with an analytical sparsity control loss (Zhuang et al., 30 Sep 2025, Yang et al., 12 Jan 2026). The router architecture is minimal, often a single linear projection per expert bank, plus optional noise for regularization.

The LD-MoLE extension, which is readily adapted to TT-LoRA experts, uses a Sparsegen projection for per-token, per-layer, and dynamically controlled expert allocation. This enables fine-grained capacity assignment: for each token feature $x_t$, the router calculates $u_t = W_\text{gate}^\top x_t$ and predicts a sparsity control $\lambda_t = f(x_t)$. The closed-form projection ensures fully differentiable, token-dependent, and layer-wise expert selection, with analytic bounds for explicit control of expert count (Zhuang et al., 30 Sep 2025). When subsumed within TT-LoRA MoE, these routing principles confer adaptive sparsity while preserving parameter and computational efficiency.

4. Layer-wise and Asymmetric Expert Allocation

Layer-wise allocation of experts is a key factor in model effectiveness. Empirical studies indicate that higher transformer layers benefit from increased expert density, as token representations grow more abstract and diverse (Gao et al., 2024, Yang et al., 12 Jan 2026). TT-LoRA MoE frameworks support explicit asymmetric expert allocation, parameterizing the number of experts per layer as a function of depth (e.g., $$N_\ell = N_\text{min} + \lfloor (N_\text{max} - N_\text{min}) (\ell / L)^\gamma \rfloor$$). This strategy outperforms both uniform and bottom-heavy allocations on downstream specialized tasks (e.g., clinical NLP) and reduces parameter redundancy without accuracy tradeoff (Yang et al., 12 Jan 2026).

A plausible implication is that layer-wise allocation complements the inherent specialization advantage of the MoE: allocating more experts higher promotes fine semantic discrimination, while lower layers—where token features are similar and expert outputs redundant—require fewer experts.

5. Parameter, Computation, and Efficiency

TT-LoRA MoE achieves significant gains in parameter and memory efficiency relative to both dense adapter and prior MoE approaches:

Adapter Type	Params per Task	Relative Size vs LoRA	Router Params
Standard LoRA	1.7M	1.0	N/A
TT-LoRA	33,920	0.02	N/A
Pfeiffer Adapter	12.6M	0.26	N/A
AdapterFusion	—	—	$10^8$
TT-LoRA MoE Router	—	—	$10^4$

(Kunwar et al., 29 Apr 2025) and (Yang et al., 12 Jan 2026) report that TT-LoRA MoE delivers comparable or superior multi-task accuracy (e.g., +4.5 points on 17 tasks vs AdapterFusion) at 0.03% of AdapterFusion’s router parameter budget. Computational costs are reduced via both TT-contraction (1.1–1.9× faster than naive adapter reconstruction) and by activating only a sparse set of experts per input.

6. Mitigating Task Interference and Catastrophic Forgetting

By freezing all expert adapters upon task completion and confining further optimization to the router, TT-LoRA MoE eliminates both inter-task interference and catastrophic forgetting (Kunwar et al., 29 Apr 2025). In domain adaptation (e.g., Med-MoE-LoRA for clinical LLMs (Yang et al., 12 Jan 2026)), architectural isolation splits experts into “base” (generalist, near-identity initialized) and “specialist” (domain-specific, randomly initialized) banks. This dual-path topology enables continual acquisition of new skills while safeguarding original model proficiency: Med-MoE-LoRA incurred near-zero forgetting on MMLU/GSM8K, whereas full fine-tuning and conventional LoRA degraded by up to 8%.

Soft or per-input router selection further avoids expert collapse, while auxiliary load-balancing losses ensure even utilization of adapter capacity.

7. Practical Applications and Empirical Performance

TT-LoRA MoE supports scalable, incremental, and modular LLM deployments:

• Multi-Task and Continual Learning: New tasks are integrated by training a single TT-LoRA adapter, without revisiting prior data or retraining the router or other experts (Kunwar et al., 29 Apr 2025).
• Resource-Constrained Environments: Each expert’s parameter cost is negligible compared to base model size, enabling storage and routing to hundreds of experts in on-device or edge settings.
• Domain-Specific Adaptation: As shown with Med-MoE-LoRA, TT-LoRA MoE achieves state-of-the-art average accuracy (e.g., $59.5$ on medical benchmarks, outperforming LoRA and vanilla MoE-LoRA) without increased inference cost or general-domain degradation (Yang et al., 12 Jan 2026).
• Mitigating Interference in Multimodal and Complex Task Blends: The dissociation of experts through TT-LoRA MoE underpins improvements in 2D/3D vision-language tasks and diverse QA benchmarks (Chen et al., 2023, Zhuang et al., 30 Sep 2025).

• “TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts” (Kunwar et al., 29 Apr 2025)
• “LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts” (Zhuang et al., 30 Sep 2025)
• “Towards Specialized Generalists: A Multi-Task MoE-LoRA Framework for Domain-Specific LLM Adaptation” (Yang et al., 12 Jan 2026)
• “Higher Layers Need More LoRA Experts” (Gao et al., 2024)
• “Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE” (Chen et al., 2023)