Papers
Topics
Authors
Recent
Search
2000 character limit reached

Adapting to Distribution Shift by Visual Domain Prompt Generation

Published 5 May 2024 in cs.CV and cs.LG | (2405.02797v1)

Abstract: In this paper, we aim to adapt a model at test-time using a few unlabeled data to address distribution shifts. To tackle the challenges of extracting domain knowledge from a limited amount of data, it is crucial to utilize correlated information from pre-trained backbones and source domains. Previous studies fail to utilize recent foundation models with strong out-of-distribution generalization. Additionally, domain-centric designs are not flavored in their works. Furthermore, they employ the process of modelling source domains and the process of learning to adapt independently into disjoint training stages. In this work, we propose an approach on top of the pre-computed features of the foundation model. Specifically, we build a knowledge bank to learn the transferable knowledge from source domains. Conditioned on few-shot target data, we introduce a domain prompt generator to condense the knowledge bank into a domain-specific prompt. The domain prompt then directs the visual features towards a particular domain via a guidance module. Moreover, we propose a domain-aware contrastive loss and employ meta-learning to facilitate domain knowledge extraction. Extensive experiments are conducted to validate the domain knowledge extraction. The proposed method outperforms previous work on 5 large-scale benchmarks including WILDS and DomainNet.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (80)
  1. A privacy-preserving unsupervised domain adaptation framework for clinical text analysis. arXiv preprint arXiv:2201.07317, 2022.
  2. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
  3. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274, 2022.
  4. Metareg: Towards domain generalization using meta-regularization. In Advances in Neural Information Processing Systems, 2018.
  5. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 38(2):550–560, 2018.
  6. The iwildcam 2021 competition dataset. arXiv preprint arXiv:2105.03494, 2021.
  7. Generalizing from several related classification tasks to a new unlabeled sample. Advances in Neural Information Processing Systems, 2011.
  8. Tinytl: Reduce memory, not parameters for efficient on-device learning. In Advances in Neural Information Processing Systems, 2020.
  9. Domain generalization by mutual-information regularization with pre-trained models. In European Conference on Computer Vision, 2022.
  10. Generalized dataweighting via class-level gradient manipulation. Advances in Neural Information Processing Systems, 2021.
  11. Gradient-based bi-level optimization for deep learning: A survey. arXiv preprint arXiv:2207.11719, 2022a.
  12. Bidirectional learning for offline infinite-width model-based optimization. NeurIPS, 2022b.
  13. Contrastive test-time adaptation. In IEEE Conference on Computer Vision and Pattern Recognition, 2022c.
  14. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In Conference on computer vision and pattern recognition, 2021.
  15. Metafscil: A meta-learning approach for few-shot class incremental learning. In Conference on computer vision and pattern recognition, 2022.
  16. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  17. Functional map of the world. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  18. Bayesian prompt learning for image-language model generalization. In Advances in Neural Information Processing Systems, 2023.
  19. Analyzing and improving representations with the soft nearest neighbor loss. In International Conference on Machine Learning, 2019.
  20. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In AAAI Conference on Artificial Intelligence, 2023.
  21. Visual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:2210.04831, 2022.
  22. Finetune like you pretrain: Improved finetuning of zero-shot vision models. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
  23. E^ 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770, 2023.
  24. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5149–5169, 2021.
  25. Diversity-aware meta visual prompting. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
  26. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, 2021.
  27. Visual prompt tuning. In European Conference on Computer Vision, 2022.
  28. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  29. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  30. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, 2021.
  31. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022.
  32. The power of scale for parameter-efficient prompt tuning. In Empirical Methods in Natural Language Processing, 2021.
  33. Learning to generalize: Meta-learning for domain generalization. In AAAI Conference on Artificial Intelligence, 2018a.
  34. Domain generalization with adversarial feature learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2018b.
  35. Rethinking out-of-distribution (ood) detection: Masked image modeling is all you need. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
  36. Deep domain generalization via conditional invariant adversarial networks. In European Confererence on Computer Vison, 2018c.
  37. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, 2020.
  38. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361, 2023.
  39. On-device training under 256kb memory. In Advances in Neural Information Processing Systems, 2022.
  40. Few-shot class-incremental learning via entropy-regularized data-free replay. In European Conference on Computer Vision, 2022a.
  41. Towards multi-domain single image dehazing via test-time training. In Conference on computer vision and pattern recognition, 2022b.
  42. Meta-auxiliary learning for future depth prediction in videos. In Winter Conference on Applications of Computer Vision, 2023.
  43. Prompt generation networks for input-based adaptation of frozen vision transformers. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
  44. Reducing domain gap via style-agnostic networks. arXiv preprint arXiv:1910.11645, 2(7):8, 2019.
  45. Blackvip: Black-box visual prompting for robust transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
  46. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  47. Black box few-shot adaptation for vision-language models. In IEEE International Conference on Computer Vision, 2023.
  48. Moment matching for multi-source domain adaptation. In IEEE International Conference on Computer Vision, 2019.
  49. From sparse to soft mixtures of experts. arXiv preprint arXiv:2308.00951, 2023.
  50. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
  51. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
  52. Extending the WILDS benchmark for unsupervised adaptation. In International Conference on Learning Representations, 2022.
  53. Test-time prompt tuning for zero-shot generalization in vision-language models. In Advances in Neural Information Processing Systems, 2022.
  54. Clipood: Generalizing clip to out-of-distributions. In International Conference on Machine Learning, 2023.
  55. Deep coral: Correlation alignment for deep domain adaptation. In European Confererence on Computer Vison Workshop, 2016.
  56. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, 2020.
  57. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.
  58. Attention is all you need. In Advances in neural information processing systems, 2017.
  59. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
  60. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, 2022a.
  61. Learning to prompt for continual learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2022b.
  62. A fine-grained analysis on distribution shift. arXiv preprint arXiv:2110.11328, 2021.
  63. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, 2022a.
  64. Robust fine-tuning of zero-shot models. In IEEE Conference on Computer Vision and Pattern Recognition, 2022b.
  65. Robust fine-tuning of zero-shot models. In IEEE Conference on Computer Vision and Pattern Recognition, 2022c.
  66. Metagcd: Learning to continually learn in generalized category discovery. In International Conference on Computer Vision, 2023.
  67. Test-time domain adaptation by learning domain-aware batch normalization. In AAAI Conference on Artificial Intelligence, 2024.
  68. Adversarial domain adaptation with domain mixup. In AAAI Conference on Artificial Intelligence, 2020a.
  69. Information leakage by model weights on federated learning. In Proceedings of the 2020 workshop on privacy-preserving machine learning in practice, 2020b.
  70. Unsupervised domain adaptation without source data by casting a bait. arXiv preprint arXiv:2010.12427, 1(2):5, 2020.
  71. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature communications, 11(1):2583, 2020.
  72. Adaptive risk minimization: Learning to adapt to domain shift. In Advances in Neural Information Processing Systems, 2021a.
  73. Domain prompt learning for efficiently adapting clip to unseen domains. arXiv preprint arXiv:2111.12853, 2021b.
  74. Youshan Zhang. A survey of unsupervised domain adaptation for visual recognition. arXiv preprint arXiv:2112.06745, 2021.
  75. Prompt vision transformer for domain generalization. arXiv preprint arXiv:2208.08914, 2022.
  76. Meta-dmoe: Adapting to domain shift by meta-distillation from mixture-of-experts. In Advances in Neural Information Processing Systems, 2022.
  77. Deep domain-adversarial image generation for domain generalisation. In AAAI Conference on Artificial Intelligence, 2020a.
  78. Learning to generate novel domains for domain generalization. In European Confererence on Computer Vison, 2020b.
  79. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022a.
  80. Conditional prompt learning for vision-language models. In IEEE Conference on Computer Vision and Pattern Recognition, 2022b.
Citations (6)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free