H2G2-Net: A Hierarchical Heterogeneous Graph Generative Network Framework for Discovery of Multi-Modal Physiological Responses
Abstract: Discovering human cognitive and emotional states using multi-modal physiological signals draws attention across various research applications. Physiological responses of the human body are influenced by human cognition and commonly used to analyze cognitive states. From a network science perspective, the interactions of these heterogeneous physiological modalities in a graph structure may provide insightful information to support prediction of cognitive states. However, there is no clue to derive exact connectivity between heterogeneous modalities and there exists a hierarchical structure of sub-modalities. Existing graph neural networks are designed to learn on non-hierarchical homogeneous graphs with pre-defined graph structures; they failed to learn from hierarchical, multi-modal physiological data without a pre-defined graph structure. To this end, we propose a hierarchical heterogeneous graph generative network (H2G2-Net) that automatically learns a graph structure without domain knowledge, as well as a powerful representation on the hierarchical heterogeneous graph in an end-to-end fashion. We validate the proposed method on the CogPilot dataset that consists of multi-modal physiological signals. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art GNNs by 5%-20% in prediction accuracy.
- Brockschmidt, M. 2020. Gnn-film: Graph neural networks with feature-wise linear modulation. In International Conference on Machine Learning, 1144–1152. PMLR.
- Relational graph attention networks. arXiv preprint arXiv:1904.05811.
- Toward Automated Instructor Pilots in Legacy Air Force Systems: Physiology-based Flight Difficulty Classification via Machine Learning. Available at SSRN 4170114.
- Fastgcn: fast learning with graph convolutional networks via importance sampling. arXiv preprint arXiv:1801.10247.
- PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. circulation, 101(23): e215–e220.
- Detecting Epileptic Seizures via Non-Uniform Multivariate Embedding of EEG Signals. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 1690–1693. IEEE.
- Optimizing non-uniform multivariate embedding for multiscale entropy analysis of complex systems. Biomedical Signal Processing and Control, 71: 103206.
- Inductive representation learning on large graphs. Advances in neural information processing systems, 30.
- Heterogeneous graph transformer. In Proceedings of the web conference 2020, 2704–2710.
- HetEmotionNet: two-stream heterogeneous graph recurrent neural network for multi-modal emotion recognition. In Proceedings of the 29th ACM International Conference on Multimedia, 1047–1056.
- Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
- Heterogeneous edge-enhanced graph attention network for multi-agent trajectory prediction. arXiv preprint arXiv:2106.07161.
- Multimodal Physiological Monitoring During Virtual Reality Piloting Tasks. PhysioNet.
- Modeling relational data with graph convolutional networks. In The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15, 593–607. Springer.
- Graph attention networks. arXiv preprint arXiv:1710.10903.
- Structural deep network embedding. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, 1225–1234.
- Heterogeneous graph attention network. In The world wide web conference, 2022–2032.
- Graph transformer networks. Advances in neural information processing systems, 32.
- Emotionmeter: A multimodal framework for recognizing human emotions. IEEE transactions on cybernetics, 49(3): 1110–1122.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.