Hierarchical Adapters

• Hierarchical adapters are modular neural modules that leverage hierarchical structures—like trees or tiers—to enable fine-grained, parameter-efficient adaptation across diverse domains and tasks.
• They implement selective parameter updates and structured aggregation to mitigate negative interference and promote positive transfer among related tasks or domains.
• Empirical results show that hierarchical adapters improve efficiency and performance in multi-domain language, vision, and speech models while reducing overall parameter count.

Hierarchical adapters are parameter-efficient, modular neural modules that enable fine-grained adaptation of large pre-trained models to multiple domains, tasks, or modalities. The key property of a hierarchical adapter system is the existence of an explicit or implicit structure—often a tree, tiered layer, or multi-level grouping—through which parameter sharing and specialization are mediated. This structure allows adapters to capture both generic and specific knowledge in a scalable manner, reducing negative interference between unrelated tasks or domains and promoting positive transfer among related ones. Hierarchical adapters have been developed across language, vision, vision-language, speech, and medical imaging models for scenarios ranging from multi-domain and multi-task learning to continual and cross-center adaptation.

1. Theoretical Basis and Structural Principles

The foundational principle of hierarchical adapter systems is that target domains, tasks, or modalities exhibit varying degrees of relatedness and share common substructures. By modeling these relationships hierarchically, adapters can be organized such that:

• Higher-level (coarse) adapters encode broadly shared patterns, updated frequently as they are active for many domains or tasks.
• Lower-level (fine) adapters capture domain- or task-specific patterns, activated only for specialized subsets.

For example, in domain adaptation for LLMs, domains (e.g., individual websites) are mapped to leaves in a tree, with internal nodes aggregating related sites (e.g., all e-commerce) (Chronopoulou et al., 2021). Each node in the tree has its own adapter parameters, and for any domain, the active adapter path is the set of nodes from root to leaf. This tree-based structural assignment encourages parameter sharing along shared subpaths and mitigates negative transfer between unrelated leaves, as unrelated domains share only root-level parameters.

Beyond explicit trees, hierarchical structures are instantiated in various forms: multi-level grouping by domain or modality (Xu et al., 18 Aug 2025), layered allocation for distinct loss objectives or representation roles (Turk et al., 21 Sep 2025), or latent semantic hierarchies in hyperbolic spaces (Zhao et al., 15 Aug 2025).

2. Algorithms and Architectures

Numerous adapter constructions implement hierarchical organization across diverse model types:

a. Tree-Structured Adapters for Domain Adaptation

For frozen transformer-based LLMs (e.g., GPT-2), each node of a domain tree is associated with a bottleneck adapter; only adapters on the path from root to the active domain are updated at train time, and their outputs are averaged at the corresponding transformer layers during inference (Chronopoulou et al., 2021). This enables $O(\log k)$ parameter scaling for $k$ domains, significantly lower than $O(k)$ for flat adapter baselines.

b. Hierarchical Regularization in VLMs

Latent Hierarchical Adapters for vision-LLMs employ a three-level semantic hierarchy—category, attribute (learnable prompts), and image instance—embedded in a hyperbolic (Poincaré ball) space (Zhao et al., 15 Aug 2025). Specialized attribute-aware refiners and hierarchical losses enforce structured alignment, supporting one-to-many mappings and improved adaptation to unseen classes.

c. Hierarchical Grouping and Merging for Continual Learning

HAM (Hierarchical Adapter Merging) introduces group-based LoRA adapters: each task starts with its own low-rank adapter, but after training, adapters are grouped by similarity, pruned for compactness, and merged hierarchically (Coleman et al., 16 Sep 2025). This enables lifelong continual learning with bounded memory—at most $M \ll N$ group adapters are retained for $N$ tasks, circumventing linear growth and reducing catastrophic forgetting.

d. Layerwise Hierarchies and Expert Mixtures

In multi-lingual discourse relation classification HiDAC, a dual adapter scheme is employed: lower layers use standard LoRA adapters under a contrastive loss, and upper layers use mixture-of-experts LoRA (MoE-LoRA) adapters, trained with cross-entropy (Turk et al., 21 Sep 2025). This split captures local representation shaping and high-level classification specialization. Similarly, Hierarchical Recurrent Adapters in speech models separate global shared controller parameters from per-task heads, amortizing most trainable parameters (Munkhdalai et al., 2024).

e. Multi-level Modality Adapters

In long video-to-text summarization, Hierarchical3D adapters perform global interaction among utterance-aligned multimodal tokens (text, audio, vision) alongside standard per-token adapters, enabling more effective fusion and global context propagation (Papalampidi et al., 2022). In medical imaging, protocol-level and center-level adapters are stacked to address both sequence-specific and scanner-specific distribution shifts, assisted by a universal adapter for unseen domains (Xu et al., 18 Aug 2025).

3. Training and Inference Procedures

The specifics of hierarchical adapter training depend on the structural embedding and modality:

• Selective Parameter Updates: For tree-based domain adaptation, only the adapters along the active path are updated for each target, with all other adapter weights and the backbone frozen. In contrast, dual-layer schemes (e.g., HiDAC) optimize distinct losses and adapters in different regions of the model.
• Regularizer Design: Hierarchical regularizers in hyperbolic space (as in LatHAdapter) enforce structured semantic proximity, while group-based merging (HAM) employs cosine similarity and pruning to consolidate adapters.
• Inference-Time Aggregation: For adaptation to unseen domains, strategies such as path-averaging over the hierarchy (e.g., two best domain paths in (Chronopoulou et al., 2021)) or on-the-fly group-merging (HAM) are applied, typically with minor computational overhead.

4. Empirical Results and Efficiency Trade-offs

Hierarchical adapters consistently outperform flat or per-task/domain adapter baselines across multiple modalities and tasks by achieving better trade-offs between performance, parameter count, and memory usage.

• In multi-domain language modeling, hierarchical adapters with a tree structure yield lower in-domain and out-of-domain perplexity compared to multi-domain or single-domain adapter baselines, with active parameters per forward pass scaling as $O(\log k)$ (Chronopoulou et al., 2021).
• In few-shot vision-language adaptation, Latent Hierarchical Adapters integrated with prompt-based baselines yield 0.7–4.7% accuracy gains on both base and novel classes with only minor increases in parameter count (Zhao et al., 15 Aug 2025).
• HAM achieves robust continual learning on up to 100 tasks by containing memory to a small, fixed number of merged adapters and displaying 15–20% less forgetting than non-hierarchical PEFT baselines (Coleman et al., 16 Sep 2025).
• HiDAC's dual-level adapter design achieves state-of-the-art multi-lingual discourse classification (67.5% vs. 66.8% for BERT 75% progressive unfreeze) while tuning only 3% of backbone parameters (Turk et al., 21 Sep 2025).
• Multimodal and cross-center hierarchical adapters (as in HierAdaptMR) yield pronounced domain generalization, e.g., 0.769→0.868 (+12.9%) improvement in SSIM for cardiac MRI reconstruction (Xu et al., 18 Aug 2025).
• Recurrent hierarchical adapters in ASR match full fine-tuning quality with just a fraction of the parameter footprint (mean WER 9.9 vs. 9.3; 0.2B vs. 232B parameters in the multi-task Euphonia benchmark) (Munkhdalai et al., 2024).

5. Limitations, Open Problems, and Generalization

Several practical and theoretical limitations remain:

• Dependency on structural priors: The quality of the hierarchy (e.g., clustering in tree-based methods) can limit effectiveness—poorly defined structures may underperform (Chronopoulou et al., 2021). In many cases, domain or task groupings must be known or estimable.
• Applicability across tasks: Most work to date has been in modeling, classification, and representation adaptation; explicit downstream utilization in generative or open-ended tasks remains less explored.
• Regularization and transfer: Hierarchical sharing naturally regulates overfitting but can propagate noise if upper layers are not properly calibrated or if negative transfer across loosely related groups persists.
• Model and adapter complexity: Mixture-of-experts and group merging methods require additional gating, grouping, and pruning steps to avoid scaling bottlenecks—especially in large continual learning settings (Coleman et al., 16 Sep 2025).

6. Extensions and Future Directions

Several clear research directions follow from the current literature:

• Learned and adaptive hierarchies: End-to-end training of both the hierarchy and adapter parameters, enabling data-driven, dynamic construction of sharing structures (Chronopoulou et al., 2021).
• Cross-modal and multi-granular hierarchies: Integrating multiple structural axes—semantic, modality, temporal—within a unified adapter framework (as in LatHAdapter and Hierarchical3D) (Zhao et al., 15 Aug 2025, Papalampidi et al., 2022).
• Synergy with other PEFT methods: Combining hierarchical adapters with techniques such as LoRA, prompt tuning, or sparse updates for enhanced flexibility (Chronopoulou et al., 2021, Turk et al., 21 Sep 2025).
• Adaptive capacity control: Dynamic growing, pruning, or reallocation of adapter banks at training or inference time for efficiency and transfer (Zhao et al., 15 Aug 2025, Coleman et al., 16 Sep 2025).
• Applications to novel tasks: Testing hierarchical adapter frameworks beyond modeling/classification, such as joint generation, open-vocabulary retrieval, or multi-modal hierarchical generation (Zhao et al., 15 Aug 2025).

7. Summary Table of Representative Hierarchical Adapter Architectures

Approach / Paper	Structure & Scope	Core Mechanism
"Efficient Hierarchical..." (Chronopoulou et al., 2021)	Tree-structured domains (LMs)	Node adapter per tree node; path-averaging; $O(\log k)$ params
"Latent Hierarchical Adapter" (Zhao et al., 15 Aug 2025)	Category–attribute–image (VLM)	Attribute prompts; hyperbolic reg.; triplet miners
"HAM: Hierarchical Adapter Merging" (Coleman et al., 16 Sep 2025)	Continual tasks (vision)	Adapter grouping & merging; LoRA; group-wise pruning
"HiDAC" (Turk et al., 21 Sep 2025)	Layerwise dual adapters (DRC)	LoRA in low layers (contrastive); MoE-LoRA upper (CE loss)
"Hierarchical3D Adapters" (Papalampidi et al., 2022)	Multimodal utterance/global (summar.)	Utterance-level fusion & interaction in enc. adapters
"HierAdaptMR" (Xu et al., 18 Aug 2025)	Protocol/center-level (MRI recon)	Stacked adapters per scanner, protocol; universal adapter
"Hierarchical Recurrent Adapter" (Munkhdalai et al., 2024)	Task-level, recurrent-shared (speech)	Global controller recur. adapter + task head per downstream

This comprehensive body of research demonstrates that hierarchical adapters can yield parameter-efficient, scalable, and robust adaptation mechanisms, adapting seamlessly to various architectures, domains, modalities, and task regimes.