APPLeNet: Visual Attention Parameterized Prompt Learning for Few-Shot Remote Sensing Image Generalization using CLIP

Abstract: In recent years, the success of large-scale vision-LLMs (VLMs) such as CLIP has led to their increased usage in various computer vision tasks. These models enable zero-shot inference through carefully crafted instructional text prompts without task-specific supervision. However, the potential of VLMs for generalization tasks in remote sensing (RS) has not been fully realized. To address this research gap, we propose a novel image-conditioned prompt learning strategy called the Visual Attention Parameterized Prompts Learning Network (APPLeNet). APPLeNet emphasizes the importance of multi-scale feature learning in RS scene classification and disentangles visual style and content primitives for domain generalization tasks. To achieve this, APPLeNet combines visual content features obtained from different layers of the vision encoder and style properties obtained from feature statistics of domain-specific batches. An attention-driven injection module is further introduced to generate visual tokens from this information. We also introduce an anti-correlation regularizer to ensure discrimination among the token embeddings, as this visual information is combined with the textual tokens. To validate APPLeNet, we curated four available RS benchmarks and introduced experimental protocols and datasets for three domain generalization tasks. Our results consistently outperform the relevant literature and code is available at https://github.com/mainaksingha01/APPLeNet

• The paper introduces APPLeNet, a novel CLIP-based method that uses visual attention parameterized prompt learning (VAPL) and multi-scale features to enhance few-shot generalization on remote sensing imagery by bridging visual and textual modalities.
• APPLeNet demonstrates superior performance on four optical RS benchmarks, consistently outperforming previous CLIP-based methods by at least 2% in mean classification scores across base-to-new class, cross-dataset, and single-source multi-target generalization tasks.
• APPLeNet offers practical utility for few-shot learning in RS applications like environmental monitoring and urban planning, while theoretically advancing feature disentanglement for improved transfer learning across domains.

Overview of APPLeNet: Visual Attention Parameterized Prompt Learning for Few-Shot Remote Sensing Image Generalization using CLIP

This paper introduces APPLeNet, a novel approach designed to enhance the generalization capability of Visual LLMs (VLMs) in the domain of Remote Sensing (RS) imagery. Leveraging the architecture of CLIP, APPLeNet addresses the limitations of zero-shot inference methods when applied to RS data, characterized by domain shifts and varying spectral and spatial features. Through a sophisticated integration of multi-scale visual features and style information, APPLeNet introduces an innovative mechanism for generating image-conditioned prompts that bridge visual and textual modalities within CLIP's framework.

APPLeNet stands out by implementing a Visual Attention Parameterized Prompts Learning (VAPL) strategy. This framework employs an attention-driven injection module that effectively combines multi-scale content features, derived from various layers of CLIP's frozen visual encoder, with domain-specific style information captured via batch feature statistics. Moreover, the model introduces an anti-correlation regularizer to ensure heterogeneous token embeddings, which significantly enhances the adaptability of the learned prompts.

The empirical validation of APPLeNet was conducted across four optical RS benchmarks, where it demonstrated superior performance in three domain generalization tasks: base-to-new class, cross-dataset, and single-source multi-target generalizations. Notably, APPLeNet consistently outperformed established CLIP-based methods, achieving improvements of at least 2% in mean classification scores across benchmarks, thereby solidifying its efficacy in addressing the unique challenges present in RS image analysis.

Numerical Results and Comparative Performance

The experimental evaluations presented in this paper show robust numerical outcomes of APPLeNet. Across multiple RS datasets, APPLeNet achieved enhanced harmonics means for base-to-new class generalization benchmarks, surpassing state-of-the-art alternatives like CoOp, CoCoOp, and ProGrad. For example, in the PatternNet dataset, APPLeNet improved the harmonic mean by achieving a score of 77.55% compared to CoOp's 74.12% and CoCoOp's 75.16%.

In the context of cross-dataset generalization, APPLeNet also demonstrated superior performance, boosting zero-shot inference accuracies on RSICD, RESISC45, and MLRSNet datasets with marked improvements in classification accuracy over baseline CLIP methods and previous methods extracting only textural information from CLIP's vision encoder. Meanwhile, in single-source multi-target settings, APPLeNet's performance remained strong even under significant domain and label shifts.

Theoretical and Practical Implications

The theoretical implications of APPLeNet lie in its ability to effectively disentangle content and style aspects of imagery, facilitating better transfer learning across varying domains. This method sets a precedent for more sophisticated integration of feature hierarchies in multi-modal learning systems, a critical step forward for AI applications that hinge on nuanced domain-adaptive models.

From a practical standpoint, APPLeNet broadens the utility of VLMs in fields where annotated data is sparse or non-generalizable due to domain-specific peculiarities. Its capability to handle few-shot learning scenarios makes it particularly appealing for applications within environmental monitoring, urban planning, and emergency management systems relying on remote sensing data.

Future Directions in AI

This study propels forward the scope of prompt learning within AI models, especially in utilizing inherent architectural layers to tackle multi-scale features in challenging data domains such as RS imagery. Future research may explore the dynamic expansion of such models to further iterate tasks across broader ML domains while reducing computational overhead.

In conclusion, APPLeNet significantly enriches the applicability of CLIP-like models to domain-sensitive tasks, marking an evolution in how AI systems interface with unstructured data across a multitude of emerging platforms.