OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning

Abstract: Prompt tuning adapts Vision-LLMs like CLIP to open-world tasks with minimal training costs. In this direction, one typical paradigm evaluates model performance separately on known classes (i.e., base domain) and unseen classes (i.e., new domain). However, real-world scenarios require models to handle inputs without prior domain knowledge. This practical challenge has spurred the development of open-world prompt tuning, which demands a unified evaluation of two stages: 1) detecting whether an input belongs to the base or new domain (P1), and 2) classifying the sample into its correct class (P2). What's more, as domain distributions are generally unknown, a proper metric should be insensitive to varying base/new sample ratios (P3). However, we find that current metrics, including HM, overall accuracy, and AUROC, fail to satisfy these three properties simultaneously. To bridge this gap, we propose OpenworldAUC, a unified metric that jointly assesses detection and classification through pairwise instance comparisons. To optimize OpenworldAUC effectively, we introduce Gated Mixture-of-Prompts (GMoP), which employs domain-specific prompts and a gating mechanism to dynamically balance detection and classification. Theoretical guarantees ensure generalization of GMoP under practical conditions. Experiments on 15 benchmarks in open-world scenarios show GMoP achieves SOTA performance on OpenworldAUC and other metrics. We release the code at https://github.com/huacong/OpenworldAUC

Overview of "OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning"

This paper introduces an innovative approach to prompt tuning in Vision-Language Models (VLMs) within the context of open-world tasks, specifically focusing on evaluating and optimizing these models using a newly proposed metric, OpenworldAUC. The work emphasizes the necessity of adapting VLMs like CLIP for scenarios where models encounter both known and unknown classes without prior domain indications, which is a significant challenge due to the unpredictable nature of class distributions in real-world applications.

Key Contributions

1. Novel Evaluation Metric: OpenworldAUC

   • Traditional metrics such as HM (Harmonic Mean), overall accuracy, and AUROC are inadequate for comprehensive evaluation in open-world settings because they either fail to simultaneously assess base-to-new detection and classification or are sensitive to class distribution changes.
   • The OpenworldAUC metric is introduced as a unified measure that evaluates both the detection of whether input belongs to the base or new domain and the subsequent classification within the correct domain.
   • OpenworldAUC uses pairwise instance comparisons, effectively bridging the gap between detection and classification evaluation, offering insensitivity to varying base/new sample ratios.

2. Gated Mixture-of-Prompts (GMoP)

   • The paper presents GMoP as an optimization framework tailored to maximize OpenworldAUC. This framework employs multiple domain-specific prompts and a gating mechanism to effectively balance detection and classification tasks.
   • GMoP divides the optimization task into specialized components, ensuring that both detection and classification objectives are adequately addressed without mutual interference, which is pivotal in open-world scenarios.

3. Comprehensive Empirical Analysis

   • Extensive experiments conducted across 15 benchmarks demonstrate the efficacy of GMoP, where it achieves state-of-the-art performance on OpenworldAUC and other standard metrics.
   • The empirical results confirm that the proposed framework not only enhances detection and classification accuracy but also maintains robustness against varying domain distributions, showcasing its practical applicability.

Theoretical Implications

The paper provides theoretical guarantees for the generalization performance of GMoP, suggesting that with a sufficiently large training set and a proper partitioning strategy, the approach can effectively adapt to new distributions encountered in open-world tasks. Moreover, the theoretical insights align with empirical observations, further validating the robustness and scalability of the proposed method.

Future Developments

The research opens promising avenues for further work in the optimization of VLMs for open-world tasks. Future research could explore extensions of OpenworldAUC to integrate additional contextual information or external knowledge, such as leveraging large language models, to better adapt VLMs in dynamic environments. Moreover, the adaptation of this approach to different architectures and its scalability across various domains remain intriguing areas for exploration.

In conclusion, "OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning" offers a significant contribution to the field by proposing a unified evaluation metric and an effective optimization framework for open-world prompt tuning in VLMs, addressing critical challenges within real-world application contexts.