Papers
Topics
Authors
Recent
Search
2000 character limit reached

Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models

Published 29 Oct 2024 in cs.CV and cs.AI | (2410.21802v2)

Abstract: Due to the impressive zero-shot capabilities, pre-trained vision-LLMs (e.g. CLIP), have attracted widespread attention and adoption across various domains. Nonetheless, CLIP has been observed to be susceptible to adversarial examples. Through experimental analysis, we have observed a phenomenon wherein adversarial perturbations induce shifts in text-guided attention. Building upon this observation, we propose a simple yet effective strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This framework incorporates two components: the Attention Refinement module and the Attention-based Model Constraint module. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness: The Attention Refinement module aligns the text-guided attention obtained from the target model via adversarial examples with the text-guided attention acquired from the original model via clean examples. This alignment enhances the model's robustness. Additionally, the Attention-based Model Constraint module acquires text-guided attention from both the target and original models using clean examples. Its objective is to maintain model performance on clean samples while enhancing overall robustness. The experiments validate that our method yields a 9.58% enhancement in zero-shot robust accuracy over the current state-of-the-art techniques across 16 datasets. Our code is available at https://github.com/zhyblue424/TGA-ZSR.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16430–16441, 2022.
  2. Food-101–mining discriminative components with random forests. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461. Springer, 2014.
  3. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pages 1877–1901, 2020.
  4. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017.
  5. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014.
  6. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223, 2011.
  7. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206–2216. PMLR, 2020.
  8. Revisiting pre-trained models for Chinese natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 657–668, 2020.
  9. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  10. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
  11. Adversarial robustness via random projection filters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4077–4086, 2023.
  12. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  13. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004.
  14. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581–595, 2024.
  15. Caltech-256 object category dataset. Technical report, Technical Report 7694, California Institute of Technology Pasadena, 2007.
  16. Calip: Zero-shot enhancement of clip with parameter-free attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 746–754, 2023.
  17. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  18. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.
  19. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340–8349, 2021.
  20. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262–15271, 2021.
  21. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021.
  22. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  23. Learning multiple layers of features from tiny images. Master’s thesis, University of Tront, 2009.
  24. Robust evaluation of diffusion-based adversarial purification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 134–144, 2023.
  25. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022.
  26. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in neural information processing systems, volume 34, pages 9694–9705, 2021.
  27. Aroid: Improving adversarial robustness through online instance-wise data augmentation. arXiv preprint arXiv:2306.07197, 2023.
  28. Data augmentation alone can improve adversarial training. In The Eleventh International Conference on Learning Representations, 2022.
  29. A novel robustness-enhancing adversarial defense approach to ai-powered sea state estimation for autonomous marine vessels. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2024.
  30. Are data-driven explanations robust against out-of-distribution data? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3821–3831, 2023.
  31. Exploring visual interpretability for contrastive language-image pre-training. arXiv preprint arXiv:2209.07046, 2022.
  32. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21152–21164, 2023.
  33. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  34. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  35. Adversarial robustness through random weight sampling. In Advances in Neural Information Processing Systems, volume 36, pages 37657–37669, 2023.
  36. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
  37. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
  38. Understanding zero-shot adversarial robustness for large-scale models. In The Eleventh International Conference on Learning Representations, 2023.
  39. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021.
  40. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582, 2016.
  41. Diffusion models for adversarial purification. In International Conference on Machine Learning, pages 16805–16827. PMLR, 2022.
  42. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722–729. IEEE, 2008.
  43. A Omkar M Parkhi. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010.
  44. Clip-guided vision-language pre-training for question answering in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5606–5611, 2023.
  45. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012.
  46. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001.07966, 2020.
  47. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  48. Denseclip: Language-guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18082–18091, 2022.
  49. Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In International Conference on Machine Learning, 2024.
  50. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  51. Adversarial training for free! In Advances in neural information processing systems, volume 32, 2019.
  52. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
  53. Class incremental learning for light-weighted networks. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
  54. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017.
  55. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pages 210–218. Springer, 2018.
  56. Augmax: Adversarial composition of random augmentations for robust training. In Advances in neural information processing systems, volume 34, pages 237–250, 2021.
  57. Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2024.
  58. Bev-clip: Multi-modal bev retrieval methodology for complex scene in autonomous driving. arXiv preprint arXiv:2401.01065, 2024.
  59. Adversarial weight perturbation helps robust generalization. In Advances in neural information processing systems, volume 33, pages 2958–2969, 2020.
  60. Adversarial purification with the manifold hypothesis. In AAAI Conference on Artificial Intelligence, 2022.
  61. Visual-language prompt tuning with knowledge-guided context optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6757–6767, 2023.
  62. Filip: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2021.
  63. Quality-agnostic image captioning to safely assist people with vision impairment. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 6281–6289, 2023.
  64. Lu Yu and Verena Rieser. Adversarial textual robustness of visual dialog. In Findings of 61st Annual Meeting of the Association for Computational Linguistics 2023, pages 3422–3438. Association for Computational Linguistics, 2023.
  65. Exploiting the semantic knowledge of pre-trained text-encoders for continual learning. arXiv preprint arXiv:2408.01076, 2024.
  66. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472–7482. PMLR, 2019.
  67. Improving the robustness of deep neural networks via stability training. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4480–4488, 2016.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Authors (3)

  1. Lu Yu 
  2. Haiyang Zhang 
  3. Changsheng Xu 

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free