Discovering Hidden Gems in Model Repositories

Abstract: Public repositories host millions of fine-tuned models, yet community usage remains disproportionately concentrated on a small number of foundation checkpoints. We investigate whether this concentration reflects efficient market selection or if superior models are systematically overlooked. Through an extensive evaluation of over 2,000 models, we show the prevalence of "hidden gems", unpopular fine-tunes that significantly outperform their popular counterparts. Notably, within the Llama-3.1-8B family, we find rarely downloaded checkpoints that improve math performance from 83.2% to 96.0% without increasing inference costs. However, discovering these models through exhaustive evaluation of every uploaded model is computationally infeasible. We therefore formulate model discovery as a Multi-Armed Bandit problem and accelerate the Sequential Halving search algorithm by using shared query sets and aggressive elimination schedules. Our method retrieves top models with as few as 50 queries per candidate, accelerating discovery by over 50x.

• The paper introduces an accelerated Sequential Halving method to efficiently identify high-performing, under-explored models.
• Empirical evaluation over 2,000 models reveals hidden gems can boost benchmark performance by up to 12.8% compared to popular models.
• It challenges reliance on popularity and documentation, advocating active evaluation for systematic model discovery.

Discovering Hidden Gems in Model Repositories: An Expert Analysis

Introduction and Problem Statement

As the growth of public model repositories such as Hugging Face accelerates into the millions of fine-tuned checkpoints, selection bottlenecks have emerged: usage patterns reveal extreme concentration on a tiny fraction of foundation models, with the overwhelming majority of models rarely or never explored. The paper "Discovering Hidden Gems in Model Repositories" (2601.22157) rigorously interrogates whether this concentration is a consequence of market efficiency (users gravitating to the best models) or a byproduct of information asymmetry and practical search limitations. Specifically, it asks: Do large-scale repositories harbor high-performing models ("hidden gems") that are systematically overlooked, and if so, how can these models be identified without prohibitive compute expenditure?

Empirical Findings: Prevalence and Distribution of Hidden Gems

Through exhaustive evaluation of over 2,000 models across four major model lineages—Qwen-3B, Qwen-7B, Mistral-7B, and Llama3.1-8B—the authors conclusively refute the efficient discovery hypothesis. Their results expose a repository landscape where the vast majority of models are unused, yet within this long tail exist models that achieve substantial performance gains over the most popular foundation checkpoints. For example, in the Llama3.1-8B family, a rarely downloaded fine-tuned model boosts math benchmark (GSM8K) performance from 83.2% to 96.0% with identical inference cost. Similar findings are replicated across downstream tasks and architectures, including dramatic improvements in coding and generalist evaluation benchmarks.

Figure 1: The Llama3.1-8B Model Tree, with node size representing downloads; gems with superior performance are circled, illustrating repository inefficiency and the extreme mismatch between popularity and capability.

Further, the search for these gems reveals their scattered, unpredictable distribution within model trees and no discernible clustering near foundation models or their direct fine-tunes. This challenges the efficacy of heuristics based on popularity, lineage centrality, or proximity, which might be expected to guide effective search.

Figure 3: Cumulative accuracy distributions across architectures and tasks; the top few models are outliers while the majority of models underperform by more than 10% relative to the best performer.

Moreover, the majority of these high-performing but obscure models lack any relevant documentation or performance metadata, rendering them invisible to documentation-driven or text-search-based discovery workflows.

Formalization and Algorithmic Innovation

Recognizing the infeasibility of exhaustive evaluation over millions of candidates, the paper introduces a principled framework for model discovery, formulating it as a Fixed-Budget Best-Arm Identification problem in the Multi-Armed Bandit (MAB) setting. Here, models are "arms," and their performance on a fixed evaluation suite is the reward.

The authors adapt Sequential Halving (SH)—an elimination-based pure-exploration MAB algorithm—with crucial domain-specific modifications. Most notably:

• Correlated Sampling: In each round, all candidate models are evaluated on the same set of queries, minimizing variance in relative model comparison and controlling for non-uniform query difficulty.
• Aggressive Early Elimination: The initial round eliminates up to 80% of low-performing models with minimal queries (e.g., 6 per model under a 10-query budget), rapidly narrowing the candidate set and efficiently reallocating computational resources.

The resulting algorithm achieves order-of-magnitude acceleration in identifying top models, consistently locating a top-3 gem with as few as 50 queries per candidate—a greater than 50× speedup over naive methods.

Figure 4: Diagrammatic overview of the proposed accelerated Sequential Halving model search algorithm.

Quantitative Evaluation

Empirical evaluation demonstrates robust superiority across all tested architectures and budgets. In the challenging low-budget regime (10 queries/model), the proposed method achieves an average rank in the top 1–5 models, while all standard baselines (uniform, UCB, Thompson sampling, vanilla SH, etc.) often fail to even outperform the widely used foundation checkpoints. With moderate query budgets (50 queries/model), gem discovery is reliable—delivering up to 12.8% accuracy gains on specific benchmarks over the popular models.

Figure 2: Coding accuracy for Qwen-3B, with gems outperforming all popular variants—yet having minimal downloads.

Distributional and Structural Observations

Visualization across model trees for different lines (Qwen, Llama, Mistral) and tasks (math, coding, multi-task) confirms a crucial, non-trivial insight: elite performers are neither localized nor easily predictable from tree topology or metadata.

Figure 5: Multiple performance-based visualizations for the Qwen-3B model tree, highlighting that gems have no consistent location or lineage pattern.

Figure 6: Analogous performance distribution visualization for the Llama3-8B tree, further confirming informational asymmetry.

Limitations and Future Directions

While this work demonstrates scalable discovery of high-performing models using active evaluation, it demands inference over all candidate models on a modest query budget—an O(number of models) cost. This motivates future research into weight space learning and representation [kahana2025can, Horwitz_2025_CVPR], which aims to learn features on model parameters for prediction and retrieval without inference, though current approaches remain limited to small-scale models and benchmarks.

Additionally, the generalization challenge remains: gems discovered for one task may not transfer to others, necessitating new evaluations per benchmark of interest.

Implications and Outlook

The findings suggest that repository centralization is decisively not a proxy for optimality: practitioners who rely on popularity metrics, lineage, or documentation for model selection are often forgoing substantial, no-cost performance improvements.

Practically, the authors' method enables systematic exploitation of latent model diversity in open repositories. Theoretically, it highlights that the full value of model populations remains largely untapped, motivating further research into query-efficient, data-free discovery—and into the structure and dynamics of model repository ecosystems.

Longer term, integration of weight-space metamodels, improved model card standards, and hybrid search that leverages both active evaluation and structural representations could further democratize repository value and close the gap between latent and realized model capability.

Conclusion

The paper provides a rigorous, quantitative indictment of current model discovery practices in large-scale repositories and delivers a practical, theoretically principled framework for surfacing elite but neglected models. Through a combination of systematic empirical audit and methodological innovation, it establishes that hidden gems are endemic, not exceptional, and unveils scalable search solutions, raising both immediate and fundamental questions about the future of open model ecosystem curation and utilization.