Unsupervised Model Tree Heritage Recovery

Abstract: The number of models shared online has recently skyrocketed, with over one million public models available on Hugging Face. Sharing models allows other users to build on existing models, using them as initialization for fine-tuning, improving accuracy, and saving compute and energy. However, it also raises important intellectual property issues, as fine-tuning may violate the license terms of the original model or that of its training data. A Model Tree, i.e., a tree data structure rooted at a foundation model and having directed edges between a parent model and other models directly fine-tuned from it (children), would settle such disputes by making the model heritage explicit. Unfortunately, current models are not well documented, with most model metadata (e.g., "model cards") not providing accurate information about heritage. In this paper, we introduce the task of Unsupervised Model Tree Heritage Recovery (Unsupervised MoTHer Recovery) for collections of neural networks. For each pair of models, this task requires: i) determining if they are directly related, and ii) establishing the direction of the relationship. Our hypothesis is that model weights encode this information, the challenge is to decode the underlying tree structure given the weights. We discover several properties of model weights that allow us to perform this task. By using these properties, we formulate the MoTHer Recovery task as finding a directed minimal spanning tree. In extensive experiments we demonstrate that our method successfully reconstructs complex Model Trees.

• The paper introduces MoTHer Recovery, which decodes neural model lineage by analyzing weight distances and directional scores.
• The methodology combines clustering and a minimum directed spanning tree algorithm to accurately map parent-child model relationships.
• Experimental validation on datasets like ViT and ecosystems such as Llama2 demonstrates high recovery accuracy, paving the way for scalable model indexing.

Unsupervised Model Tree Heritage Recovery: A Comprehensive Analysis

Overview of Model Tree Heritage Recovery

The paper "Unsupervised Model Tree Heritage Recovery" (2405.18432) introduces a novel method for decoding the hereditary relationships between neural network models through their weights. The crux of this approach, termed Model Tree Heritage Recovery (MoTHer Recovery), stems from an observation that model weights encode lineage information similar to Darwin's tree of life concept. This method holds promise for elucidating model authorship attribution and potentially indexing the AI model universe akin to internet search engines.

Methodology: Decoding Model Relationships

MoTHer Recovery aims to construct a directed graph, termed Model Tree, that maps the parent-child relationships between models. It leverages two primary aspects of model weights: weight distance and directional weight score. Weight distances, computed as differences of outlier values within model layers, correlate with node distance within model trees. The directional weight score is calculated using the kurtosis of weight distributions, evolving monotonically during training. These metrics facilitate not only the establishment of relatedness but also directionality of model derivations.

Figure 1: Directional Weight Score: The plot illustrates the monotonic change during generalization and specialization stages.

Recovery Process and Algorithm

To recover a Model Graph, MoTHer employs clustering to identify different Model Trees based on pairwise weight distances. Subsequently, a minimum directed spanning tree (MDST) algorithm is applied to a distance matrix that combines weight distance and directional matrix elements. This approach ensures that the hereditary nature and direction of model relationships are captured accurately.

Figure 2: Recovering a Simplified Model Graph: Demonstrated process of determining parent-child relationships using weight distance and directional scores.

Experimental Validation

The paper tests MoTHer Recovery on a ViT Tree Heritage Recovery dataset and real-world model ecosystems such as Llama2 and Stable Diffusion. Results showed promising accuracy in reconstructing model hierarchies. Notably, the LoRA fine-tuning variant demonstrated perfect recovery accuracy due to its distinct low-rank matrix updates. In-the-wild ecosystems further validated the approach, reconciling complex hierarchies with previously established standards.

Figure 3: ViT Tree Heritage Recovery (VTHR) Dataset Overview: Visualization of a single Model Graph within the dataset.

Implications and Future Directions

The potential applications of MoTHer Recovery extend beyond immediate model attribution to longer-term innovations such as building comprehensive model databases that account for training histories and optimization strategies. Future challenges include scaling the Model Graph recovery process to accommodate the web-scale model repositories, necessitating efficient data handling and real-time updates as model landscapes evolve.

Figure 4: MoTHer Recovery Overview: Schematic representation of the recovery algorithm and Model Graph creation process.

Conclusion

"Unsupervised Model Tree Heritage Recovery" provides a methodical and technically robust approach for understanding neural model lineage through unsupervised learning processes. The combination of empirical evaluation and algorithmic advancements posits MoTHer Recovery as a foundational step towards achieving comprehensive interpretable AI model mappings. The work invites further exploration into scalable model graph recovery techniques and the integration of metadata to enrich the AI ecosystem.

In summary, this paper establishes crucial methodologies for model lineage discovery, offering significant insights into neural network genealogy and setting the stage for future research in AI model indexing and interpretability.