InCTRLv2: Generalist Residual Models for Few-Shot Anomaly Detection and Segmentation

Abstract: While recent anomaly detection (AD) methods have made substantial progress in recognizing abnormal patterns within specific domains, most of them are specialist models that are trained on large training samples from a specific target dataset, struggling to generalize to unseen datasets. To address this limitation, the paradigm of Generalist Anomaly Detection (GAD) has emerged in recent years, aiming to learn a single generalist model to detect anomalies across diverse domains without retraining. To this end, this work introduces InCTRLv2, a novel few-shot Generalist Anomaly Detection and Segmentation (GADS) framework that significantly extends our previously proposed GAD model, InCTRL. Building on the idea of learning in-context residuals with few-shot normal examples to detect anomalies as in InCTRL, InCTRLv2 introduces two new, complementary perspectives of anomaly perception under a dual-branch framework. This is accomplished by two novel modules upon InCTRL: i) Discriminative Anomaly Score Learning (DASL) with both normal and abnormal data in the main branch, which learns a semantic-guided abnormality and normality space that supports the classification of query samples from both the abnormality and normality perspectives; and ii) One-class Anomaly Score Learning (OASL) using only the normal data, which learns generalized normality patterns in a semantic space via an auxiliary branch, focusing on detecting anomalies through the lens of normality solely. Both branches are guided by rich visual-text semantic priors encoded by large-scale vision-LLMs. Together, they offer a dual semantic perspective for AD: one emphasizes normal-abnormal discriminations, while the other emphasizes normality-deviated semantics. Extensive experiments on ten AD datasets demonstrate that InCTRLv2 achieves SotA performance in both anomaly detection and segmentation tasks across various settings.

• The paper introduces a dual-branch architecture combining discriminative and one-class semantic guidance for few-shot anomaly detection and segmentation.
• It leverages CLIP-based text-image alignment and in-context residual learning for robust image- and patch-level anomaly mapping.
• Empirical results on 10 datasets show state-of-the-art improvements in AUROC and anomaly localization across industrial, medical, and semantic domains.

InCTRLv2: Generalist Residual Models for Few-Shot Anomaly Detection and Segmentation

Problem Formulation and Motivation

Anomaly detection (AD) in computer vision has traditionally relied on specialist models trained for each data domain, which impedes scalability and cross-domain generalization, especially in scenarios where acquiring large-scale labeled data is infeasible. The emerging paradigm of Generalist Anomaly Detection and Segmentation (GADS) addresses this limitation by seeking a single model capable of few-shot adaptation and generalized inference across diverse domains. InCTRLv2 advances this paradigm through an integrated dual-branch architecture, leveraging semantic priors from vision-LLMs and in-context residual learning, enabling robust anomaly detection and localization from limited normal samples across industrial, medical, and semantic domains.

Architectural Overview

InCTRLv2 extends the InCTRL framework with two main innovations: (1) a Discriminative Anomaly Score Learning (DASL) branch that exploits both normal and abnormal samples, augmented with semantic guidance via CLIP-based text-image feature alignment, and (2) a One-class Anomaly Score Learning (OASL) branch that exclusively learns from normal samples to distill domain-invariant normality priors. This dual-branch structure integrates three complementary anomaly signals—residual, discriminative, and one-class semantic guidance—into a unified learning and inference mechanism.

Figure 1: A conceptual illustration of the GADS paradigm, where the model leverages auxiliary pretraining and a small number of normal in-context prompts for zero-shot application to unseen domains.

Figure 2: Training pipeline of InCTRLv2, depicting the dual-branch architecture: the main branch’s DASL module learns a joint normal-abnormal semantic space, while the auxiliary OASL branch models generalized normality.

The framework is instantiated atop a pre-trained CLIP ViT backbone, employing few-shot normal images from the target domain as in-context prompts. During inference, these prompts guide residual scoring at both the image and patch levels, while semantic anomaly maps are synthesized via text prompt prototypes representing normality and abnormality.

Methodology

In-Context Residual Learning

The main mechanism for domain transfer is the computation of image-level and patch-level residuals between a query and few-shot normal samples. Image-level differences are based on the class token embedding, processed and compared via an adapter for anomaly-sensitive representation. Patch-level residuals exploit hierarchical ViT patch tokens, measuring nearest-neighbor cosine distances over the prompt set to generate dense anomaly maps with both global and fine-grained cues.

Figure 3: The DASL module’s mechanisms for image-level and hierarchical patch-level in-context learning, producing residual features and dense anomaly maps.

Semantic-Guided Joint Learning

• Discriminative Anomaly Score Learning (DASL): This branch aligns visual features with hand-crafted textual prototypes for both normal and abnormal classes, delivering both global scores and pixel-level anomaly maps. Lightweight adapters harmonize CLIP’s representations with anomaly detection objectives and cross-layer fusion aggregates multi-resolution cues. The final anomaly scores are fused from both semantic discrimination and max residuals.
• One-class Anomaly Score Learning (OASL): Trained solely on normal data, OASL distills stable normality semantics via text-guided feature alignment, yielding robust signals for anomaly correction in new domains and reducing false positives caused by out-of-distribution domain shifts.

At inference, the pixel-level outputs from the discriminative and one-class branches are combined to generate the final anomaly map, with a hyperparameter $\beta$ balancing the two sources to optimize localization fidelity.

Experimental Results

Comprehensive experiments on ten real-world datasets—including VisA, MVTecAD, BraTS, and more—demonstrate the superiority of InCTRLv2 over both classic and recent VLM-based FSAD methods. The model consistently achieves the best or second-best performance in AUROC and AP metrics for both image-level and pixel-level AD across industrial, medical, and semantic domains, especially in the challenging one- and two-shot settings.

Notable empirical claims include:

• Up to 3.9% improvement in AUROC for industrial, 3.7% for medical, and 5.7% for semantic datasets compared to the strongest baseline.
• Enhanced anomaly localization (PRO metrics): up to 1.5% (industrial) and 16.3% (medical) over prior state-of-the-art approaches.
• Strong qualitative improvements in anomaly map coherence and suppression of false alarms, particularly in medical imaging and semantic outlier settings.

  Figure 4: Visualization of anomaly maps generated by the DASL module alone versus the refined output of InCTRLv2, highlighting improved localization and reduced false positives due to the OASL correction.

  

  Figure 5: Comparative visualization of pixel-level anomaly detection by different GADS methods; InCTRLv2 achieves enhanced specificity and continuity across diverse contexts.

Ablation and Analysis

Targeted ablation studies confirm the additive benefit of combining semantic-guided objectives ($\mathcal{L}_{I}$ and $\mathcal{L}_{P}$ in DASL) and the normality-guided loss ($\mathcal{L}_{OASL}$). Notably, OASL’s impact is most pronounced in reducing pixel-wise false positives and improving mask coherence, underscoring its value for robust domain-general anomaly localization.

Figure 6: Sensitivity of image-level and pixel-level AUROC to the fusion parameters $\alpha$ and $\beta$, demonstrating optimal balance when combining global and local anomaly cues and highlighting the stable contribution from OASL.

Efficiency benchmarks reveal that InCTRLv2 achieves a favorable trade-off, matching or surpassing other VLM-based methods in parameter count and inference speed, making it practical for deployment in real-time or resource-constrained environments.

Implications and Future Directions

InCTRLv2 contributes a principled architecture for open-world AD, demonstrating that joint modeling of discriminative and normality-centric semantics, grounded in vision-language priors, greatly enhances generalization and robustness in few-shot and cross-domain regimes. The method does not require domain-specific retraining or prompt engineering, indicating potential for large-scale unsupervised deployment in manufacturing, medical screening, and scientific imaging scenarios, where labeled data is scarce or privacy-restricted.

Theoretically, this work motivates further research into dual-perspective anomaly modeling, particularly the extension of semantic-guided one-class learning for unsupervised distributional robustness. Practically, integrating richer language supervision, leveraging foundation models with multi-modal alignment, and exploring fast prompt selection for dynamic environments are promising avenues.

Conclusion

InCTRLv2 establishes new benchmarks for few-shot, generalist anomaly detection and segmentation, validating the efficacy of dual-branch semantic-guided residual learning on diverse, open-world image domains. The complementary use of discriminative and one-class branches delivers robust detection and interpretable localization in few-shot settings, advancing the deployment of GADS frameworks in safety-critical and annotation-scarce applications.