AMoE: Agglomerative Mixture-of-Experts Vision Foundation Model

Abstract: Vision foundation models trained via multi-teacher distillation offer a promising path toward unified visual representations, yet the learning dynamics and data efficiency of such approaches remain underexplored. In this paper, we systematically study multi-teacher distillation for vision foundation models and identify key factors that enable training at lower computational cost. We introduce Agglomerative Mixture-of-Experts Vision Foundation Models (AMoE), which distill knowledge from SigLIP2 and DINOv3 simultaneously into a Mixture-of-Experts student. We show that (1) our Asymmetric Relation-Knowledge Distillation loss preserves the geometric properties of each teacher while enabling effective knowledge transfer, (2) token-balanced batching that packs varying-resolution images into sequences with uniform token budgets stabilizes representation learning across resolutions without sacrificing performance, and (3) hierarchical clustering and sampling of training data--typically reserved for self-supervised learning--substantially improves sample efficiency over random sampling for multi-teacher distillation. By combining these findings, we curate OpenLVD200M, a 200M-image corpus that demonstrates superior efficiency for multi-teacher distillation. Instantiated in a Mixture-of-Experts. We release OpenLVD200M and distilled models.

• The paper introduces a multi-teacher distillation framework that aggregates SigLIP2 and DINOv3 into a Mixture-of-Experts foundation model with novel architectural innovations.
• It leverages token-balanced batching and Asymmetric Relational Knowledge Distillation to ensure stable learning and preserve the geometric structure of representations.
• Empirical results demonstrate state-of-the-art performance on global and dense vision benchmarks, highlighting enhanced data efficiency and improved handling of long-tail categories.

Agglomerative Mixture-of-Experts Vision Foundation Model: AMoE

Introduction

The paper "AMoE: Agglomerative Mixture-of-Experts Vision Foundation Model" (2512.20157) introduces a methodology for data- and compute-efficient multi-teacher distillation in vision foundation models (VFMs). The authors address challenges in aggregating complementary supervision signals from distinct, state-of-the-art vision teacher models. AMoE distills SigLIP2 and DINOv3 into a single Mixture-of-Experts (MoE) student backbone, introducing architectural and algorithmic innovations for stable, scalable, and efficient learning of unified visual representations. This essay details AMoE’s design, its technical contributions, empirical performance, and implications for further research.

Architecture and Distillation Framework

AMoE leverages a Mixture-of-Experts student in which 28 experts (with sparse, task-adaptive routing) receive knowledge distilled from multiple frozen teacher models—SigLIP2 (for image-text alignment) and DINOv3 (for dense, geometric features). Each input image is processed through both teachers and the student; the student’s backbone representations are projected into the embedding spaces of both teachers using teacher-specific adaptor heads. Register tokens, as in DINOv3, are used to enhance dense feature representations.

The distillation objective is multi-faceted, simultaneously aligning the student’s patch, global (CLS), and register embeddings with the corresponding teacher outputs, with projection heads ensuring effective transfer to each teacher’s domain.

Figure 1: The AMoE pipeline, showing the aggregation of signals from SigLIP2 and DINOv3 into a Mixture-of-Experts student via specialized projection heads and multiple supervised representations.

Key Algorithmic Contributions

Token-Balanced Batching

A critical bottleneck in large-scale vision model training is the variance in the number of tokens introduced by processing images at their native resolutions. AMoE addresses this with token-balanced batching: sequences are packed up to a fixed token budget, and FlexAttention masks are applied to prevent inter-image attention, ensuring computational efficiency and stable learning regardless of input resolution. Losses are normalized by per-image token count to prevent high-resolution samples from dominating gradient updates.

Figure 2: Token-balanced batching upscales throughput and prevents degradation of low-resolution image representations.

Asymmetric Relational Knowledge Distillation (ARKD)

While pointwise alignment of teacher-student representations is standard, it fails to guarantee preservation of the embedding geometry. AMoE extends relational knowledge distillation with ARKD: instead of uniformly minimizing the difference in pairwise distances between all samples, ARKD only penalizes student embeddings for 'shrinking' (becoming too close) or 'expanding' (becoming too far) if the corresponding teacher pair lies below or above the median teacher distance, respectively. This asymmetric design preserves clustering structure and significantly improves alignment, especially for teachers whose geometry is not enforced by their own objectives (e.g., Dense ViTs like DINOv3).

Figure 3: ARKD regularizes the evolving geometry in the student representation space, especially for teachers with weak global alignment objectives.

Curated Data: OpenLVD200M

To enhance data efficiency, AMoE employs a dataset curation pipeline based on hierarchical clustering. OpenLVD200M, constructed from the LAION and DFN datasets, is a 200M-image corpus sampled to maximize semantic and long-tail concept coverage while minimizing redundancy. This approach outperforms random sampling by a substantial margin, especially for rare categories and fine-grained classification tasks.

Figure 4: Hierarchical clustering and sampling ensures a balanced representation of both head and tail concepts.

Empirical Results and Analysis

AMoE achieves state-of-the-art global and dense vision performance on benchmarks including ImageNet, Caltech101, CUB-200, Food-101, FGVC-Aircraft, and segmentation tasks (ADE20k, Cityscapes, PASCAL-VOC).

On global image-text classification and kNN (macro-average), AMoE (with only 200M curated images) robustly outperforms RADIOv2.5-H, which was trained on $\sim1$B images—a $\sim4.7\times$ reduction in total tokens. The model achieves macro-averaged image-text classification of 84.13 and kNN at 87.44, compared to 82.26/84.42 for RADIOv2.5-H. For challenging long-tail categories (e.g., FGVC-Aircraft), AMoE demonstrates a relative gain of nearly 13 percentage points.

Figure 5: CKA similarity analysis reveals explicit teacher specialization in AMoE’s experts across depth, underlining effective multi-teacher aggregation.

Token-balanced batching and ARKD both show measurable improvements. ARKD preserves kNN clustering quality, addressing the standard RKD’s degeneracy on local structure, while maintaining strong global alignment, as reflected by substantial boosts in image-text retrieval Recall@1 on MSCOCO5k and Flickr30k. AMoE’s expert routing further enables adaptive transfer specialization, with some experts specializing in high-norm teacher layers.

Theoretical and Practical Implications

Data efficiency and resource scaling:

AMoE’s results underscore that a careful curation of training data, in combination with hierarchical distillation objectives and batching, can approach or surpass models trained with four to five times the compute, eliminating redundancy and improving access to rare concepts.

Geometry preservation and ensemble synergy:

ARKD’s asymmetric design offers a principled mechanism for transferring structural properties across models. Empirically, ensembling teacher heads via entropy-weighted fusion yields gains absent in baselines, confirming that complementary alignment is achieved in AMoE.

Expert specialization analytics:

Layer-wise CKA analysis shows that the MoE architecture is not merely a computational trick but establishes hierarchical specialization, with selected experts aligning preferentially to different teachers across depth and feature scale.

Grounded Vision-LLMs:

Early-fusion VLMs initialized with distilled AMoE experts, especially with Gram anchoring to preserve dense feature quality, show strong grounding accuracy—even with limited downstream annotation—indicating practical utility for applications requiring high data efficiency and composability.

Limitations and Future Directions

The AMoE framework, though substantially more efficient, still depends on large-scale semi/noisy-labeled data and compute for clustering. Extensions to domain-adaptive distillation, integration with language signals beyond SigLIP2, and scaling to even more heterogeneous sets of teachers remain unexplored. The principled design of ARKD could be expanded to cross-modal or temporal geometry preservation, potentially unifying vision and language MoEs at scale. Further, a more granular analysis of the impact of router calibration and expert sparsity could yield insights relevant to mixture-of-expert optimization in high-dimensional settings.

Conclusion

AMoE offers an architectural and algorithmic recipe for scalable, multi-domain vision backbone distillation. By integrating token-balanced batching, asymmetric relational distillation, and hierarchical data curation, it achieves significant gains in efficiency, diversity, and transfer quality. The framework affords a methodological foundation for future VFMs bridging dense, global, and cross-modal signals, supporting research into modular, compositional, and expert-specialized architectures for visual understanding.