Papers
Topics
Authors
Recent
Search
2000 character limit reached

GO4Align: Group Optimization for Multi-Task Alignment

Published 9 Apr 2024 in cs.LG and cs.CV | (2404.06486v2)

Abstract: This paper proposes \textit{GO4Align}, a multi-task optimization approach that tackles task imbalance by explicitly aligning the optimization across tasks. To achieve this, we design an adaptive group risk minimization strategy, comprising two techniques in implementation: (i) dynamical group assignment, which clusters similar tasks based on task interactions; (ii) risk-guided group indicators, which exploit consistent task correlations with risk information from previous iterations. Comprehensive experimental results on diverse benchmarks demonstrate our method's performance superiority with even lower computational costs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9(8):1295, 2020.
  2. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481–2495, 2017.
  3. 970 million druglike small molecules for virtual screening in the chemical universe database GDB-13. J. Am. Chem. Soc., 131:8732, 2009.
  4. Multi-task learning in natural language processing: An overview. arXiv preprint arXiv:2109.09138, 2021.
  5. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pp.  794–803. PMLR, 2018.
  6. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. arXiv preprint arXiv:2010.06808, 2020.
  7. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3213–3223, 2016.
  8. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp.  1407–1416. PMLR, 2018.
  9. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428, 2019.
  10. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34:27503–27516, 2021.
  11. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  3205–3214, 2019.
  12. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pp.  270–287, 2018.
  13. Learning to branch for multi-task learning. In International Conference on Machine Learning, pp.  3854–3863. PMLR, 2020.
  14. Learning with whom to share in multi-task feature learning. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp.  521–528, 2011.
  15. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  7482–7491, 2018.
  16. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  17. Review on determining number of cluster in k-means clustering. International Journal, 1(6):90–95, 2013.
  18. Kokkinos, I. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  6129–6138, 2017.
  19. Genetic k-means algorithm. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 29(3):433–439, 1999.
  20. In defense of the unitary scalarization for deep multi-task learning. Advances in Neural Information Processing Systems, 35:12169–12183, 2022.
  21. A closer look at loss weighting in multi-task learning. arXiv preprint arXiv:2111.10603, 2021a.
  22. Reasonable effectiveness of random weighting: A litmus test for multi-task learning. arXiv preprint arXiv:2111.10603, 2021b.
  23. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878–18890, 2021.
  24. Famo: Fast adaptive multitask optimization. Advances in Neural Information Processing Systems, 33, 2023.
  25. Towards impartial multi-task learning. In International Conference on Learning Representations, 2020.
  26. Adversarial multi-task learning for text classification. arXiv preprint arXiv:1704.05742, 2017.
  27. End-to-end multi-task learning with attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  1871–1880, 2019.
  28. Auto-lambda: Disentangling dynamic task relationships. arXiv preprint arXiv:2202.03091, 2022.
  29. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  30. Learning multiple tasks with multilinear relationship networks. Advances in neural information processing systems, 30, 2017.
  31. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5334–5343, 2017.
  32. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  1851–1860, 2019.
  33. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3994–4003, 2016.
  34. Self-paced multitask learning with shared knowledge. arXiv preprint arXiv:1703.00977, 2017.
  35. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
  36. Multi-task learning as a bargaining game. arXiv preprint arXiv:2202.01017, 2022.
  37. Task weighting in meta-learning with trajectory optimisation. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
  38. Conditionally adaptive multi-task learning: Improving transfer learning in nlp using fewer parameters & less data. In International Conference on Learning Representations, 2020.
  39. Multi-task learning as multi-objective optimization. arXiv preprint arXiv:1810.04650, 2018.
  40. Independent component alignment for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  20083–20093, 2023.
  41. Unsupervised k-means clustering algorithm. IEEE access, 8:80716–80727, 2020.
  42. Multi-task reinforcement learning with context-based representations. arXiv preprint arXiv:2102.06177, 2021.
  43. Efficient and effective multi-task grouping via meta learning on task combinations. Advances in Neural Information Processing Systems, 35:37647–37659, 2022.
  44. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pp.  9120–9132. PMLR, 2020.
  45. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33:8728–8740, 2020.
  46. Clustering learning tasks and the selective cross-task transfer of knowledge. In Learning to learn, pp.  235–257. Springer, 1998.
  47. Branched multi-task networks: deciding what layers to share. arXiv preprint arXiv:1904.02920, 2019.
  48. Multi-task learning for dense prediction tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  49. Do current multi-task optimization methods in deep learning even help? Advances in Neural Information Processing Systems, 35:13597–13609, 2022.
  50. Robust task grouping with representative tasks for clustered multi-task learning. In Proceedings of the 25th ACM SIGKDD International conference on knowledge discovery & data mining, pp.  1408–1417, 2019.
  51. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782, 2020.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Summarize

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 4 tweets with 12 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free