Papers
Topics
Authors
Recent
Search
2000 character limit reached

Neural Architecture Retrieval

Published 16 Jul 2023 in cs.AI | (2307.07919v2)

Abstract: With the increasing number of new neural architecture designs and substantial existing neural architectures, it becomes difficult for the researchers to situate their contributions compared with existing neural architectures or establish the connections between their designs and other relevant ones. To discover similar neural architectures in an efficient and automatic manner, we define a new problem Neural Architecture Retrieval which retrieves a set of existing neural architectures which have similar designs to the query neural architecture. Existing graph pre-training strategies cannot address the computational graph in neural architectures due to the graph size and motifs. To fulfill this potential, we propose to divide the graph into motifs which are used to rebuild the macro graph to tackle these issues, and introduce multi-level contrastive learning to achieve accurate graph representation learning. Extensive evaluations on both human-designed and synthesized neural architectures demonstrate the superiority of our algorithm. Such a dataset which contains 12k real-world network architectures, as well as their embedding, is built for neural architecture retrieval.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:1611.02167, 2016.
  2. End-to-end object detection with transformers. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pp.  213–229. Springer, 2020.
  3. Pre-training tasks for embedding-based large-scale retrieval. arXiv preprint arXiv:2002.03932, 2020.
  4. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  1294–1303, 2019.
  5. Contrastive neural architecture search with neural architecture comparators. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9502–9511, 2021.
  6. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  7. Adversarially robust neural architectures. arXiv preprint arXiv:2009.00902, 2020.
  8. Handling long-tailed feature distribution in addernets. Advances in Neural Information Processing Systems, 34:17902–17912, 2021.
  9. Improving lightweight addernet via distillation from ℓ2subscriptℓ2\ell_{2}roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT to ℓ1subscriptℓ1\ell_{1}roman_ℓ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-norm. IEEE Transactions on Image Processing, 2023.
  10. Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  1761–1770, 2019.
  11. Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326, 2020.
  12. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  13. Breaking the curse of space explosion: Towards efficient nas with curriculum search. In International Conference on Machine Learning, pp. 3822–3831. PMLR, 2020.
  14. Towards accurate and compact architectures via neural architecture transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6501–6516, 2021.
  15. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
  16. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  1580–1589, 2020.
  17. Ghostnets on heterogeneous devices via cheap operations. International Journal of Computer Vision, 130(4):1050–1069, 2022.
  18. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  19. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  20. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  7132–7141, 2018.
  21. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265, 2019.
  22. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4700–4708, 2017.
  23. Embedding-based retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.  2553–2561, 2020.
  24. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
  25. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.
  26. Spatial-channel token distillation for vision mlps. In International Conference on Machine Learning, pp. 12685–12695. PMLR, 2022a.
  27. Neural architecture search via proxy validation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7595–7610, 2022b.
  28. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493, 2015.
  29. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018.
  30. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  10012–10022, 2021.
  31. Disturbance-immune weight sharing for neural architecture search. Neural Networks, 144:553–564, 2021.
  32. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, pp.  4780–4789, 2019.
  33. Vitas: Vision transformer architecture search. In European Conference on Computer Vision, pp.  139–157. Springer Nature Switzerland Cham, 2022.
  34. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  1–9, 2015.
  35. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10781–10790, 2020.
  36. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  9627–9636, 2019.
  37. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347–10357. PMLR, 2021.
  38. Unas: Differentiable architecture search meets reinforcement learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  11266–11275, 2020.
  39. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  40. Graph attention networks. arXiv preprint arXiv:1710.10903, 2017.
  41. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
  42. Learning versatile filters for efficient convolutional neural networks. Advances in Neural Information Processing Systems, 31, 2018.
  43. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pp. 7105–7114. PMLR, 2019.
  44. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812–5823, 2020.
  45. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
  46. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  6848–6856, 2018.
  47. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  8697–8710, 2018.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up