FlowerFormer: Empowering Neural Architecture Encoding using a Flow-aware Graph Transformer
Abstract: The success of a specific neural network architecture is closely tied to the dataset and task it tackles; there is no one-size-fits-all solution. Thus, considerable efforts have been made to quickly and accurately estimate the performances of neural architectures, without full training or evaluation, for given tasks and datasets. Neural architecture encoding has played a crucial role in the estimation, and graphbased methods, which treat an architecture as a graph, have shown prominent performance. For enhanced representation learning of neural architectures, we introduce FlowerFormer, a powerful graph transformer that incorporates the information flows within a neural architecture. FlowerFormer consists of two key components: (a) bidirectional asynchronous message passing, inspired by the flows; (b) global attention built on flow-based masking. Our extensive experiments demonstrate the superiority of FlowerFormer over existing neural encoding methods, and its effectiveness extends beyond computer vision models to include graph neural networks and auto speech recognition models. Our code is available at http://github.com/y0ngjaenius/CVPR2024_FLOWERFormer.
- Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375, 2018.
- On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006.05205, 2020.
- Residual gated graph convnets. arXiv preprint arXiv:1711.07553, 2017.
- Graph-based neural architecture search with operation embeddings. In ICCV, 2021.
- Contrastive neural architecture search with neural architecture comparators. In CVPR, 2021a.
- Not all operations contribute equally: Hierarchical operation-adaptive predictor for neural architecture search. In ICCV, 2021b.
- Graph propagation transformer for graph representation learning. In IJCAI, 2023.
- Nasgem: Neural architecture search via graph embedding method. In AAAI, 2021.
- Nas-bench-201: Extending the scope of reproducible neural architecture search. In ICLR, 2020.
- A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
- Deep residual learning for image recognition. In CVPR, 2016.
- Densely connected convolutional networks. In CVPR, 2017.
- Global self-attention as a replacement for graph convolution. In KDD, 2022.
- Pharmacophoric-constrained heterogeneous graph transformer model for molecular property prediction. Communications Chemistry, 6(1):60, 2023.
- Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016.
- Rethinking graph transformers with spectral attention. In NeurIPS, 2021.
- Progressive neural architecture search. In ECCV, 2018.
- Tnasp: A transformer-based nas predictor with a self-evolution framework. In NeurIPS, 2021.
- Neural architecture optimization. NeurIPS, 2018.
- Semi-supervised neural architecture search. In NeurIPS, 2020.
- Transformers over directed acyclic graphs. In NeurIPS, 2023.
- Graph inductive biases in transformers without message passing. In ICML, 2023.
- Nas-bench-asr: Reproducible neural architecture search for speech recognition. In ICLR, 2020.
- Graphit: Encoding graph structure in transformers. arXiv preprint arXiv:2106.05667, 2021.
- Modular graph transformer networks for multi-label image classification. In AAAI, 2021.
- A generic graph-based neural architecture encoding scheme for predictor-based nas. In ECCV, 2020.
- Ta-gates: An encoding scheme for neural network architectures. In NeurIPS, 2022.
- Chemistry guided molecular graph transformer. In NeurIPS 2022 Workshop: AI for Science: Progress and Promises, 2022.
- Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947, 2019.
- Igformer: Interaction graph transformer for skeleton-based human interaction recognition. In ECCV, 2022.
- Nas-bench-graph: Benchmarking graph neural architecture search. In NeurIPS, 2022.
- Recipe for a general, powerful, scalable graph transformer. In NeurIPS, 2022.
- Self-supervised graph transformer on large-scale molecular data. In NeurIPS, 2020.
- Pranab Kumar Sen. Estimates of the regression coefficient based on kendall’s tau. Journal of the American statistical association, 63(324):1379–1389, 1968.
- Bridging the gap between sample-based and one-shot neural architecture search with bonas. NeurIPS, 2020.
- Benchmarking graphormer on large-scale molecular modeling datasets. arXiv preprint arXiv:2203.04810, 2022.
- Directed acyclic graph neural networks. In ICLR, 2021.
- Attention is all you need. In NeurIPS, 2017.
- Alphax: exploring neural architectures with deep neural networks and monte carlo tree search. arXiv preprint arXiv:1903.11059, 2019.
- Neural predictor for neural architecture search. In ECCV, 2020.
- Bananas: Bayesian optimization with neural architectures for neural architecture search. In AAAI, 2021a.
- How powerful are performance predictors in neural architecture search? In NeurIPS, 2021b.
- Cate: Computation-aware neural architecture encoding with transformers. In ICML, 2021.
- Nar-former: Neural architecture representation learning towards holistic attributes prediction. In CVPR, 2023.
- Nas-bench-101: Towards reproducible neural architecture search. In ICML, 2019.
- Do transformers really perform badly for graph representation? In NeurIPS, 2021.
- Nas-bench-1shot1: Benchmarking and dissecting one-shot neural architecture search. In ICLR, 2019.
- Surrogate nas benchmarks: Going beyond the limited search spaces of tabular nas benchmarks. In ICLR, 2022.
- A graph-transformer for whole slide image classification. IEEE transactions on medical imaging, 41(11):3003–3015, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.