Papers
Topics
Authors
Recent
Search
2000 character limit reached

Anytime Neural Architecture Search on Tabular Data

Published 15 Mar 2024 in cs.LG | (2403.10318v2)

Abstract: The increasing demand for tabular data analysis calls for transitioning from manual architecture design to Neural Architecture Search (NAS). This transition demands an efficient and responsive anytime NAS approach that is capable of returning current optimal architectures within any given time budget while progressively enhancing architecture quality with increased budget allocation. However, the area of research on Anytime NAS for tabular data remains unexplored. To this end, we introduce ATLAS, the first anytime NAS approach tailored for tabular data. ATLAS introduces a novel two-phase filtering-and-refinement optimization scheme with joint optimization, combining the strengths of both paradigms of training-free and training-based architecture evaluation. Specifically, in the filtering phase, ATLAS employs a new zero-cost proxy specifically designed for tabular data to efficiently estimate the performance of candidate architectures, thereby obtaining a set of promising architectures. Subsequently, in the refinement phase, ATLAS leverages a fixed-budget search algorithm to schedule the training of the promising candidates, so as to accurately identify the optimal architecture. To jointly optimize the two phases for anytime NAS, we also devise a budget-aware coordinator that delivers high NAS performance within constraints. Experimental evaluations demonstrate that our ATLAS can obtain a good-performing architecture within any predefined time budget and return better architectures as and when a new time budget is made available. Overall, it reduces the search time on tabular data by up to 82.75x compared to existing NAS approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. Automl: Train high-quality custom machine learning models with minimal effort and machine learning expertise. 2023. URL https://cloud.google.com/automl.
  2. Zero-cost proxies for lightweight NAS. In International Conference on Learning Representations, ICLR, 2021.
  3. A convergence theory for deep learning via over-parameterization. In Proceedings of the International Conference on Machine Learning, ICML, pp.  242–252, 2019.
  4. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, volume 35, pp.  6679–6687, 2021.
  5. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, NeurIPS, 2019.
  6. Designing neural network architectures using reinforcement learning. In International Conference on Learning Representations, ICLR, 2017.
  7. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
  8. PASHA: efficient HPO and NAS with progressive resource allocation. In International Conference on Learning Representations, ICLR, 2023.
  9. Deep neural networks and tabular data: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  10. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, ICLR, 2020.
  11. Arm-net: Adaptive relation modeling network for structured data. In International Conference on Management of Data, 2021.
  12. Tabcaps: A capsule neural network for tabular data classification with bow routing. In International Conference on Learning Representations, ICLR, 2023a.
  13. Trompt: Towards a better deep neural network for tabular data. In Proceedings of the International Conference on Machine Learning, ICML, volume 202, pp.  4392–4434, 2023b.
  14. Neural architecture search on imagenet in four GPU hours: A theoretically inspired perspective. In International Conference on Learning Representations, ICLR, 2021.
  15. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. In International Conference on Computer Vision, ICCV, pp.  12219–12228, 2021.
  16. Blossom: an anytime algorithm for computing optimal decision trees. In Proceedings of the International Conference on Machine Learning, ICML, volume 202, pp.  7533–7562, 2023.
  17. Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations, ICLR, 2020.
  18. Agebo-tabular: joint neural architecture and hyperparameter search with autotuned data-parallel training for tabular data. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2021.
  19. BOHB: robust and efficient hyperparameter optimization at scale. In Proceedings of the International Conference on Machine Learning, ICML, volume 80, pp.  1436–1445, 2018.
  20. Do we need hundreds of classifiers to solve real world classification problems? The journal of machine learning research, 15(1):3133–3181, 2014.
  21. Probabilistic matrix factorization for automated machine learning. In Advances in Neural Information Processing Systems, NeurIPS, pp.  3352–3361, 2018.
  22. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp.  249–256, 2010.
  23. Revisiting deep learning models for tabular data. In Advances in Neural Information Processing Systems, NeurIPS, pp.  18932–18943, 2021.
  24. On embeddings for numerical features in tabular deep learning. In Advances in Neural Information Processing Systems, NeurIPS, 2022.
  25. Complexity of linear regions in deep networks. In Proceedings of the International Conference on Machine Learning, ICML, volume 97, pp.  2596–2604, 2019.
  26. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.  1026–1034, 2015.
  27. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
  28. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, NeurIPS, 2018.
  29. Non-stochastic best arm identification and hyperparameter optimization. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS, volume 51, pp.  240–248, 2016.
  30. Well-tuned simple nets excel on tabular datasets. In Advances in Neural Information Processing Systems, NeurIPS, 2021.
  31. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  32. Self-normalizing neural networks. In Advances in Neural Information Processing Systems, NeurIPS, 2017.
  33. Nas-bench-suite-zero: Accelerating research on zero cost proxies. In Advances in Neural Information Processing Systems, NeurIPS, 2022.
  34. Efficient backprop. In Neural networks: Tricks of the trade, pp.  9–50. Springer, 2002.
  35. Meta-prediction model for distillation-aware NAS on unseen datasets. In International Conference on Learning Representations, ICLR, 2023.
  36. Snip: single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, ICLR, 2019.
  37. Transfer learning with deep tabular models. In International Conference on Learning Representations, ICLR, 2023.
  38. Zico: Zero-shot NAS via inverse coefficient of variation on gradients. In International Conference on Learning Representations, ICLR, 2023.
  39. Best practices for scientific research on neural architecture search. arXiv preprint arXiv:1909.02453, 2019.
  40. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Conference on Computer Vision and Pattern Recognition, CVPR, pp.  82–92, 2019a.
  41. DARTS: differentiable architecture search. In International Conference on Learning Representations, ICLR, 2019b.
  42. The expressive power of neural networks: A view from the width. In Advances in Neural Information Processing Systems, NeurIPS, pp.  6231–6239, 2017.
  43. Improving data analytics with fast and adaptive regularization. IEEE Trans. Knowl. Data Eng., 33, 2021.
  44. Regularized pairwise relationship based analytics for structured data. Proceedings of the ACM SIGMOD International Conference on Management of Data, 2023.
  45. Neural architecture search without training. In Proceedings of the International Conference on Machine Learning, ICML, volume 139, pp.  7588–7598, 2021.
  46. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Proceedings of the Workshop on Automatic Machine Learning, volume 64, 2016.
  47. Efficient neural architecture search via parameter sharing. In Proceedings of the International Conference on Machine Learning, ICML, volume 80, pp.  4092–4101, 2018.
  48. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312, 2019.
  49. Retrieval & interaction machine for tabular data prediction. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  1379–1389, 2021.
  50. On the expressive power of deep neural networks. In Proceedings of the International Conference on Machine Learning, ICML, volume 70, pp.  2847–2854, 2017.
  51. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, volume 33, pp.  4780–4789, 2019a.
  52. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, pp.  4780–4789, 2019b.
  53. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1–34, 2021.
  54. Transfer NAS with meta-learned bayesian surrogates. In International Conference on Learning Representations, ICLR, 2023.
  55. Trainability of relu networks and data-dependent initialization. Journal of Machine Learning for Modeling and Computing, 1, 2020.
  56. NASI: label- and data-agnostic neural architecture search at initialization. In International Conference on Learning Representations, ICLR, 2022a.
  57. Unifying and boosting gradient-based training-free neural architecture search. In Advances in Neural Information Processing Systems, NeurIPS, 2022b.
  58. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.09777, 11, 2020.
  59. Pruning neural networks without any data by iteratively conserving synaptic flow. In Advances in Neural Information Processing Systems, NeurIPS, 2020.
  60. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801.05787, 2018.
  61. Blockswap: Fisher-guided block substitution for network compression on a budget. In International Conference on Learning Representations, ICLR, 2020.
  62. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, ICLR, 2020a.
  63. CL4CTR: A contrastive learning framework for CTR prediction. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp.  805–813, 2023a.
  64. Trainability preserving neural pruning. In International Conference on Learning Representations, ICLR, 2023.
  65. N-wl: A new hierarchy of expressivity for graph neural networks. In International Conference on Learning Representations, ICLR, 2023b.
  66. Textnas: A neural architecture search space tailored for text representation. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, volume 34, pp.  9242–9249, 2020b.
  67. How powerful are performance predictors in neural architecture search? In Advances in Neural Information Processing Systems, NeurIPS, pp.  28454–28469, 2021.
  68. Neural architecture search: Insights from 1000 papers. CoRR, abs/2301.08727, 2023.
  69. A survey on neural architecture search. arXiv preprint arXiv:1905.01392, 2019.
  70. Fives: Feature interaction via edge search for large-scale tabular data. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp.  3795–3805, 2021.
  71. NAS evaluation is frustratingly hard. In International Conference on Learning Representations, ICLR, 2020.
  72. OBOE: collaborative filtering for automl model selection. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2019.
  73. Tabnas: Rejection sampling for neural architecture search on tabular datasets. In Advances in Neural Information Processing Systems, NeurIPS, 2022.
  74. Revisiting training-free nas metrics: An efficient training-based method. In Winter Conference on Applications of Computer Vision, WACV, pp.  4751–4760, 2023.
  75. Nas-bench-101: Towards reproducible neural architecture search. In Proceedings of the International Conference on Machine Learning, ICML, volume 97, pp.  7105–7114, 2019.
  76. When expressivity meets trainability: Fewer than $n$ neurons can work. In Advances in Neural Information Processing Systems, NeurIPS, pp.  9167–9180, 2021.
  77. Retiarii: A deep learning exploratory-training framework. In Operating Systems Design and Implementation, OSDI, pp.  919–936, 2020.
  78. Naspipe: high performance and reproducible pipeline parallel supernet training via causal synchronous parallelism. In International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS, pp.  374–387, 2022.
  79. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, ICLR, 2017.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free