How Powerful are Performance Predictors in Neural Architecture Search?

Abstract: Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks. To reduce this extreme computational cost, dozens of techniques have since been proposed to predict the final performance of neural architectures. Despite the success of such performance prediction methods, it is not well-understood how different families of techniques compare to one another, due to the lack of an agreed-upon evaluation metric and optimization for different constraints on the initialization time and query time. In this work, we give the first large-scale study of performance predictors by analyzing 31 techniques ranging from learning curve extrapolation, to weight-sharing, to supervised learning, to "zero-cost" proxies. We test a number of correlation- and rank-based performance measures in a variety of settings, as well as the ability of each technique to speed up predictor-based NAS frameworks. Our results act as recommendations for the best predictors to use in different settings, and we show that certain families of predictors can be combined to achieve even better predictive power, opening up promising research directions. Our code, featuring a library of 31 performance predictors, is available at https://github.com/automl/naslib.

• The paper conducts the first large-scale evaluation of 31 performance predictors, comparing methods from zero-cost proxies to learning curve extrapolation.
• It uses diverse benchmarks and metrics like Pearson and Spearman correlations to rigorously assess predictor performance across various NAS settings.
• The findings recommend ensemble approaches, notably the OMNI predictor, to enhance NAS efficiency by combining the strengths of different techniques.

Analyzing the Efficacy of Performance Predictors in Neural Architecture Search

This paper undertakes a comprehensive examination of performance predictors used in Neural Architecture Search (NAS), a crucial area in automating neural network design. The primary aim is to understand how different performance prediction methods compare and how they can be optimally utilized across various settings. The authors conduct an extensive evaluation of 31 predictors ranging from learning curve extrapolation to zero-cost proxies.

Key Contributions

The study provides several significant contributions:

1. Large-scale Evaluation: This is the first extensive study that compares a broad spectrum of performance prediction techniques in NAS, covering methods such as supervised learning models, zero-cost proxies, and weight-sharing mechanisms.
2. Diverse Metrics and Benchmarks: The paper evaluates predictors using correlation and rank-based performance measures across various NAS benchmarks and constraints. This includes NAS-Bench-101, NAS-Bench-201, and DARTS, among others, and employs metrics like Pearson and Spearman correlations.
3. Recommendations for Predictors: The findings offer insights into which predictors perform best under specific conditions, considering factors like initialization and query times. The results suggest that zero-cost and model-based predictors can be effectively combined to enhance performance prediction.
4. OMNI Predictor: The introduction of the OMNI predictor showcases the potential of combining different families of predictors to achieve superior predictive performance, leveraging the complementary strengths of each method.

Experimental Insights

The experimental setup is meticulous, considering various initializations, query times, and datasets. The findings indicate:

• Zero-cost Predictors: Surprisingly effective even when compared to more complex model-based methods with longer initialization times. Jacobian covariance consistently shows strong performance.
• Model-based Methods: Show varying degrees of success depending on the dataset and search space complexity. Methods like NGBoost and SemiNAS perform particularly well when combined with other predictors.
• Hybrid Approaches: Predictors like LcSVR and variants of learning curve extrapolation demonstrate notable improvements in certain settings, though model-based approaches often overshadow them in others.
• Search Space Complexity: NAS-Bench-101, with its greater diversity, benefits from predictors using specific encodings like the path encoding, indicating the importance of search space characteristics in evaluating predictor performance.

Implications and Future Directions

The implications of this work are manifold for both practical NAS applications and theoretical insights:

• Enhanced NAS Efficiency: By identifying the strengths of different predictors, this study aids in reducing the computational overhead associated with NAS, making it more accessible and efficient.
• Future Research Directions: The complementary nature of different predictors suggests promising avenues for developing more sophisticated ensemble techniques, potentially leading to breakthroughs in the accuracy and speed of NAS.
• Practical Considerations: The release of the code and library of predictors serves as a valuable resource for the research community, enabling further experimentation and refinement of NAS methodologies.

Conclusion

This paper provides a critical analysis of performance predictors in NAS, offering both a comprehensive evaluation and practical recommendations to enhance the efficiency and effectiveness of NAS techniques. The results advocate for a nuanced approach in selecting predictors, tailored to the specific constraints and goals of the NAS task at hand. As NAS continues to evolve, this work lays the groundwork for future innovations in predictor-based automation of machine learning model design.