Maximizing Information in Domain-Invariant Representation Improves Transfer Learning
Abstract: The most effective domain adaptation (DA) technique involves the decomposition of data representation into a domain-independent representation (DIRep) and a domain-dependent representation (DDRep). A classifier is trained by using the DIRep on the labeled source images. Since the DIRep is domain invariant, the classifier can be "transferred" to make predictions for the target domain with no (or few) labels. However, information useful for classification in the target domain can "hide" in the DDRep. Current DA algorithms, such as Domain-Separation Networks (DSN), do not adequately address this issue. DSN's weak constraint to enforce the orthogonality of DIRep and DDRep allows this hiding effect and can result in poor performance. To address this shortcoming, we develop a new algorithm wherein a stronger constraint is imposed to minimize the information content in DDRep to create a DIRep that retains relevant information about the target labels and, in turn, results in a better invariant representation. By using synthetic datasets, we show explicitly that depending on the initialization, DSN, with its weaker constraint, can lead to sub-optimal solutions with poorer DA performance. In contrast, our algorithm is robust against such perturbations. We demonstrate the equal-or-better performance of our approach against DSN and other recent DA methods by using several standard benchmark image datasets. We further highlight the compatibility of our algorithm with pre-trained models for classifying real-world images and showcase its adaptability and versatility through its application in network intrusion detection.
- Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898–916, 2010.
- Domain separation networks. Advances in neural information processing systems, 29, 2016.
- Improved techniques for adversarial discriminative domain adaptation. IEEE Transactions on Image Processing, 29:2622–2637, 2019.
- Adversarial-learned loss for domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 3521–3528, 2020.
- Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee, 2009.
- Informative feature disentanglement for unsupervised domain adaptation. IEEE Transactions on Multimedia, 24:2407–2421, 2021.
- Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096–2030, 2016.
- Deep reconstruction-classification networks for unsupervised domain adaptation. In European conference on computer vision, pp. 597–613. Springer, 2016.
- Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
- Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
- Variational interaction information maximization for cross-domain disentanglement. Advances in Neural Information Processing Systems, 33:22479–22491, 2020.
- Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. PMLR, 2015.
- Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- Transferable adversarial training: A general approach to adapting deep classifiers. In International Conference on Machine Learning, pp. 4013–4022. PMLR, 2019.
- Deep unsupervised domain adaptation: a review of recent advances and perspectives. APSIPA Transactions on Signal and Information Processing, 11(1), 2022.
- Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
- Reading digits in natural images with unsupervised feature learning. 2011.
- Domain agnostic learning with disentangled representations. In International Conference on Machine Learning, pp. 5102–5112. PMLR, 2019.
- Pedro O Pinheiro. Unsupervised domain adaptation with similarity learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8004–8013, 2018.
- Adapting visual category models to new domains. In Computer Vision–ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp. 213–226. Springer, 2010.
- Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8503–8512, 2018.
- Preparing network intrusion detection deep learning models with minimal data using adversarial domain adaptation. In Proceedings of the 15th ACM Asia Conference on Computer and Communications Security, pp. 127–140, 2020.
- Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167–7176, 2017.
- Deep visual domain adaptation: A survey. Neurocomputing, 312:135–153, 2018.
- Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1426–1435, 2019.
- Transfer adaptation learning: A decade survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
- Youshan Zhang. A survey of unsupervised domain adaptation for visual recognition. arXiv preprint arXiv:2112.06745, 2021.
- A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1), 2020.
- Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pp. 289–305, 2018.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.