Efficient Training of Agglomerative Multi-Teacher VFMs

Determine whether agglomerative Vision Foundation Models trained via multi-teacher distillation can be trained more efficiently within a standardized framework while preserving or improving representational quality.

Background

Agglomerative Vision Foundation Models combine complementary capabilities from multiple teacher models (e.g., self-supervised and vision–language encoders) via multi-teacher distillation. While promising, prior approaches have been computationally expensive and data-hungry, with complex handling for varying resolutions and multiple loss functions.

This work proposes components such as token-balanced batching, hierarchical data curation (OpenLVD200M), and Asymmetric Relational Knowledge Distillation to improve efficiency. However, the broader question of achieving efficient training in a standardized framework while maintaining or improving representational quality is explicitly raised as an open question, motivating the study.