InLINE: Inner-Layer Information Exchange for Multi-task Learning on Heterogeneous Graphs
Abstract: Heterogeneous graph is an important structure for modeling complex relational data in real-world scenarios and usually involves various node prediction tasks within a single graph. Training these tasks separately may neglect beneficial information sharing, hence a preferred way is to learn several tasks in a same model by Multi-Task Learning (MTL). However, MTL introduces the issue of negative transfer, where the training of different tasks interferes with each other as they may focus on different information from the data, resulting in suboptimal performance. To solve the issue, existing MTL methods use separate backbones for each task, then selectively exchange beneficial features through interactions among the output embeddings from each layer of different backbones, which we refer to as outer-layer exchange. However, the negative transfer in heterogeneous graphs arises not simply from the varying importance of an individual node feature across tasks, but also from the varying importance of inter-relation between two nodes across tasks. These inter-relations are entangled in the output embedding, making it difficult for existing methods to discriminate beneficial information from the embedding. To address this challenge, we propose the Inner-Layer Information Exchange (InLINE) model that facilitate fine-grained information exchanges within each graph layer rather than through output embeddings. Specifically, InLINE consists of (1) Structure Disentangled Experts for layer-wise structure disentanglement, (2) Structure Disentangled Gates for assigning disentangled information to different tasks. Evaluations on two public datasets and a large industry dataset show that our model effectively alleviates the significant performance drop on specific tasks caused by negative transfer, improving Macro F1 by 6.3% on DBLP dataset and AUC by 3.6% on the industry dataset compared to SoA methods.
- Identifying beneficial task relations for multi-task learning in deep neural networks. In Lapata, M.; Blunsom, P.; and Koller, A., eds., Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, 164–169. Association for Computational Linguistics.
- Graph Representation Learning for Multi-Task Settings: a Meta-Learning Approach. In International Joint Conference on Neural Networks, IJCNN 2022. IEEE.
- Caruana, R. 1997. Multitask learning. Machine learning.
- Crawshaw, M. 2020. Multi-Task Learning with Deep Neural Networks: A Survey. CoRR, abs/2009.09796.
- Cross-heterogeneity Graph Few-shot Learning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, 420–429. ACM.
- DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks. In KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 279–288. ACM.
- Homophily-oriented heterogeneous graph rewiring. In Proceedings of the ACM Web Conference 2023, 511–522.
- Adaptive Transfer Learning on Graph Neural Networks. In KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 565–574. ACM.
- Multi-task learning on graphs with node and graph level labels. In NeurIPS Workshop on Graph Representation Learning.
- An Attention-Based Graph Neural Network for Heterogeneous Structural Learning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, 4132–4139. AAAI Press.
- Heterogeneous Graph Transformer. In WWW ’20: The Web Conference 2020, 2704–2710. ACM / IW3C2.
- GDA-HIN: A Generalized Domain Adaptive Model across Heterogeneous Information Networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, 4054–4058. ACM.
- Multi-task Self-supervised Graph Neural Networks Enable Stronger Task Generalization. In The Eleventh International Conference on Learning Representations, ICLR 2023. OpenReview.net.
- Adam: A Method for Stochastic Optimization. In Bengio, Y.; and LeCun, Y., eds., 3rd International Conference on Learning Representations, ICLR 2015.
- Boosting Multitask Learning on Graphs through Higher-Order Task Affinities. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023. ACM.
- Boosting Multitask Learning on Graphs through Higher-Order Task Affinities. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM.
- Hierarchical Multi-Task Graph Recurrent Network for Next POI Recommendation. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM.
- Graph Star Net for Generalized Multi-Task Learning. CoRR, abs/1906.12330.
- Are we really making much progress?: Revisiting, benchmarking and refining heterogeneous graph neural networks. In KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 1150–1160. ACM.
- Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM.
- Cross-Stitch Networks for Multi-task Learning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016. IEEE Computer Society.
- Sluice networks: Learning what to share between loosely related tasks. CoRR, abs/1705.08142.
- Multi-Task Learning of Graph-based Inductive Representations of Music Content. In Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021.
- Modeling Relational Data with Graph Convolutional Networks. In The Semantic Web - 15th International Conference, ESWC 2018, volume 10843 of Lecture Notes in Computer Science, 593–607. Springer.
- Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations. In RecSys 2020: Fourteenth ACM Conference on Recommender Systems, 269–278. ACM.
- Heterogeneous graph masked autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence.
- Graph Attention Networks. In International Conference on Learning Representations.
- M2GRL: A Multi-task Multi-view Graph Representation Learning Framework for Web-scale Recommender Systems. In KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM.
- Single-shot Feature Selection for Multi-task Recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 341–351. ACM.
- Multi-Task Deep Recommender Systems: A Survey. CoRR, abs/2302.03525.
- Heterogeneous Network Representation Learning: A Unified Framework With Survey and Benchmark. IEEE Trans. Knowl. Data Eng., 34(10): 4854–4873.
- GraphSAINT: Graph Sampling Based Inductive Learning Method. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net.
- MugRep: A Multi-Task Hierarchical Graph Representation Learning Framework for Real Estate Appraisal. In KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM.
- A Modulation Module for Multi-task Learning with Applications in Image Retrieval. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I, Lecture Notes in Computer Science. Springer.
- A Comprehensive Survey on Transfer Learning. Proc. IEEE, 109(1): 43–76.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.