Anomaly-Aware Vision-Language Adapters for Zero-Shot Anomaly Detection

Abstract: Zero-shot anomaly detection aims to identify defects in unseen categories without target-specific training. Existing methods usually apply the same feature transformation to all samples, treating normal and anomalous data uniformly despite their fundamentally asymmetric distributions, compact normals versus diverse anomalies. We instead exploit this natural asymmetry by proposing AVA-DINO, an anomaly-aware vision-language adaptation framework with dual specialized branches for normal and anomalous patterns that adapt frozen DINOv3 visual features. During training on auxiliary data, the two branches are learned jointly with a text-guided routing mechanism and explicit routing regularization that encourages branch specialization. At test time, only the input image and fixed, predefined language descriptions are used to dynamically combine the two branches, enabling an asymmetric activation. This design prevents degenerate uniform routing and allows context-specific feature transformations. Experiments across nine industrial and medical benchmarks demonstrate state-of-the-art performance, achieving 93.5% image-AUROC on MVTec-AD and strong cross-domain generalization to medical imaging without domain-specific fine-tuning. https://github.com/aqeeelmirza/AVA-DINO

• The paper introduces a dual-adapter framework, AVA-DINO, that uses specialized normal and anomaly branches to recalibrate DINOv3 features for improved zero-shot anomaly detection.
• It leverages text-guided dynamic routing and routing regularization to fuse visual and semantic cues, ensuring balanced pixel-wise and image-level performance.
• Experimental evaluations on nine industrial and medical datasets show significant improvements in AUROC and F1 scores compared to previous approaches.

Anomaly-Aware Vision-Language Adapters for Zero-Shot Anomaly Detection

Motivation and Problem Setting

Zero-shot anomaly detection (ZSAD) remains a central challenge for industrial quality inspection and medical imaging due to the inherent difficulty in collecting comprehensive defect samples and the pronounced semantic asymmetry between normal and anomalous distributions. Conventional ZSAD methods typically perform adaptation via a single feature transformation pathway, applying uniform adjustments to both normal and anomalous samples without regard to their contrasting statistical characteristics. This uniformity inhibits the specialization needed for robust generalization, especially when anomalies exhibit wide diversity while normals are tightly clustered.

AVA-DINO Framework

The proposed AVA-DINO architecture introduces dual specialized adaptation routes to address feature transformation asymmetry, leveraging frozen DINOv3 Vision Transformer features and semantic supervision from CLIP text embeddings.

Figure 1: Illustration of uniform vs dual adaptation routes, depicting mixed features from a single adapter compared to specialized representations and routing weights from AVA-DINO adapters.

During training, AVA-DINO employs:

• Dual-branch adapters: Normal and anomaly-specific lightweight modules recalibrate features extracted from DINOv3, producing context-optimized embeddings.
• Text-guided dynamic routing: Routing decisions are computed via temperature-scaled softmax on cosine similarities between the image CLS token and projected text representations (via a learnable matrix), such as "a photo of perfect hazelnut" vs "a photo of damaged hazelnut".
• Routing regularization: Explicitly encourages pathway specialization by penalizing deviation from ideal routing assignments on auxiliary labeled data, which prevents routing collapse and fosters interpretable context selection.
• Weighted fusion: Output embeddings are a weighted sum of normal- and anomaly-adapted features, plus the original visual features, ensuring smooth interpolation and gradient flow.

The full pipeline integrates multi-scale patch tokens and CLS tokens from the DINOv3 backbone, processes them through dual adapters, applies text-driven fusion, and outputs both pixel-wise anomaly maps and image-level scores.

Figure 2: AVA-DINO architecture showing frozen DINOv3 feature extraction, dual-branch adapters, CLS-driven text-based routing, and fusion for anomaly map and prediction generation.

Training and Inference

The training objective combines pixel-level segmentation losses (Focal + Dice) and global image-level supervision with the routing regularization term. Routing regularization aligns weights with auxiliary ground-truth labels, ensuring that normals activate the normal branch ($w_n \to 1$) and anomalies activate the anomaly branch ($w_a \to 1$). At inference, routing weights are computed autonomously using test images and fixed textual prompts; adaptation proceeds without further tuning or domain-specific supervision.

Experimental Evaluation

AVA-DINO was evaluated on nine industrial and medical datasets, including MVTec-AD, ViSA, BTAD, KSDD2, MPDD, MVTec-AD2 (industrial) and Kvasir, CVC-ColonDB, CVC-ClinicDB (medical). All models were trained on auxiliary data and tested via strict zero-shot protocols.

Quantitative results show AVA-DINO achieves:

• 93.5% Image-AUROC (I-AUC) and 50.6% Pixel-F1 on MVTec-AD, outperforming prior works by 1.2-3.2 points across metrics.
• Strong generalization to medical benchmarks, e.g., 90.7% Pixel-AUROC and 57.2% Pixel-F1 on CVC-ClinicDB, with +7.8/+15.1 improvement over the second-best methods.

These improvements hold across both domains, regardless of normal/anomaly distribution widths or defect types. AVA-DINO maintains balanced performance rather than optimizing a single metric, and is robust to domain shifts, as confirmed by fewer performance fluctuations than WinCLIP when transferring to medical imaging.

Figure 3: Anomaly localization results with ground-truth boundaries (green); AVA-DINO yields sharp, precise masks on industrial and medical samples, unlike diffuse activations from earlier approaches.

Ablation studies substantiated the necessity of dual-branch adapters, text-guided routing, and routing regularization. Dual adapters contribute +0.8% ABC improvement, text-guided routing adds further gains, and explicit regularization enables true specialization, with routing weights averaging $w_n = 0.83$ for normal and $w_a = 0.79$ for anomaly samples.

Multi-layer feature extraction from DINOv3 outperformed single-layer extraction by +1.7% P-AUC, implicating the importance of both fine-grained and semantic information for localization.

Practical and Theoretical Implications

AVA-DINO's architectural design—separating normal and anomaly adaptation branches and incorporating text-guided dynamic routing backed by explicit regularization—exemplifies a new paradigm for vision-language anomaly detection systems. Practically, this enables robust, domain-agnostic deployment in scenarios where exhaustive defect collection is unfeasible, including industrial automation and medical imaging. Theoretically, the results underscore the superior generalization capacity of asymmetric adaptation, especially for settings entailing distributional mismatch and unseen semantic categories.

The interpretable nature of routing weights lays groundwork for explainability in vision-language anomaly detection. This strategy can be readily extended to few-shot domains, video data, and other foundation model adaptation tasks where context awareness and semantic divergence are crucial.

Conclusion

AVA-DINO demonstrates that dual-branch anomaly-aware adaptation, coordinated by text-guided routing and explicit regularization, is more effective than uniform adaptation for zero-shot anomaly detection in both industrial and medical domains. Its consistent superior performance and robust cross-domain generalization highlight the utility of context specialization in vision-language frameworks. Future work should explore extensions to few-shot and temporal settings and investigate generalization to new modalities and more varied text prompts.