Visual-Text Anomaly Mapper (VTAM)

• Visual-Text Anomaly Mapper (VTAM) is a multimodal framework that fuses multi-scale visual features with semantic text for anomaly detection, localization, and retrieval.
• It employs innovative modules like Local-Global Hybrid, dual-gated calibration, and region-aware encoders to generate precise anomaly maps and scores.
• VTAM underpins diverse applications from industrial inspections to fine-grained video anomaly detection, significantly enhancing accuracy and interpretability.

The Visual-Text Anomaly Mapper (VTAM) denotes a class of architectures and modules designed for joint visual-textual anomaly detection, localization, and retrieval. VTAM instantiates as a core building block across multiple state-of-the-art frameworks for tasks ranging from text-based person anomaly retrieval to industrial visual inspection and fine-grained video anomaly detection. Its principal innovation is the formulation of cross-modal alignment procedures that tightly couple multi-scale visual features with semantically conditioned text representations, often via local-global, multi-expert, or dual-gated calibration mechanisms. VTAM modules can output either scalar anomaly scores, dense pixel-level anomaly maps, or drive further high-level reasoning in a downstream vision-language network.

1. Architectural Compositions and Core Modules

VTAM underpins diverse applications with distinct, but related, module architectures:

• In text-based person anomaly retrieval, VTAM comprises a Local-Global Hybrid Perspective (LHP) module, a Unified Image-Text (UIT) model integrating four loss types (Image-Text Contrastive, Image-Text Matching, Masked Image Modeling, Masked Language Modeling), an LHP-guided candidate selection mechanism, and an iterative ensemble re-ranking strategy. The LHP module stochastically balances fine-grained (cropped pedestrian) and global (scene context) visual cues when producing embeddings for contrastive alignment with text queries (Nguyen et al., 27 Nov 2025).
• In the context of zero-shot industrial anomaly detection, the SSVP framework integrates VTAM as a dual-gated mixture-of-experts atop multi-scale visual features. Here, VTAM receives multi-level local feature maps and a global semantic embedding (from Hierarchical Semantic-Visual Synergy, HSVS), along with a vision-conditioned text prompt (from the Vision-Conditioned Prompt Generator, VCPG). Pixelwise anomaly probabilities are refined through global scale and local spatial gates, then aggregated into an anomaly heatmap and a global-local fused anomaly score (Fu et al., 14 Jan 2026).
• For fine-grained video anomaly detection, VTAM consists of an Anomaly Heatmap Decoder (AHD) that computes pixel-wise visual-text alignment via cosine similarity between projected multi-scale features and sentence-level class embeddings. This is extended by a Region-aware Anomaly Encoder (RAE), which distills spatiotemporal anomaly activations into region- and motion-aware prompts, further enhancing text-guided anomaly reasoning by large vision-LLMs (Gu et al., 1 Nov 2025).

2. Mathematical Formulation and Cross-Modal Fusion

At the algorithmic core, VTAM modules universally deploy cross-modal alignment between visual and textual modalities via high-dimensional embedding spaces. Key mathematical components include:

• Local-Global Mixture (LHP): The LHP module transforms each image $I$ into a local crop $I_\mathrm{loc}$ and a global context $I_\mathrm{glob}$. Embeddings are fused as

$f_i = w f^\mathrm{loc} + (1-w) f^\mathrm{glob}$

where $f^\mathrm{loc} = E_i(I_\mathrm{loc})$ and $f^\mathrm{glob} = E_i(I_\mathrm{glob})$ with $w$ either randomly sampled or learned (Nguyen et al., 27 Nov 2025).

• Contrastive Loss (ITC): Visual and text embeddings $(f_i, f_t)$ are trained to maximize cosine similarity for matching pairs and minimize it for mismatches via a temperature-scaled softmax across batch pairs:

$$S_{i2t}(i,j) = \frac{\exp(\mathrm{cos}(f_i^i, f_t^j)/\tau)}{\sum_{k=1}^N \exp(\mathrm{cos}(f_i^i, f_t^k)/\tau)}$$

$$\mathcal{L}_\mathrm{cl} = -\frac{1}{2}\left(\sum_i \log S_{i2t}(i,i) + \sum_j \log S_{t2i}(j,j)\right)$$

• Dual-Gated Calibration (for dense anomaly maps): In SSVP, for each spatial location $(h, w)$ in feature map $l$,

$$\mathcal{M}^{(l)}_{h,w} = \frac{V_\mathrm{syn}^{\mathrm{local},(l)}(h,w) \cdot T_\mathrm{final}^\top}{\|V_\mathrm{syn}^{\mathrm{local},(l)}(h,w)\| \|T_\mathrm{final}\|}$$

A per-pixel probability is then,

$$\mathcal{P}_\mathrm{anom}^{(l)}(h,w) = \frac{\exp(\mathcal{M}^{(l)}_\mathrm{anom}(h,w)/\tau)}{\sum_{c \in \{\mathrm{norm},\mathrm{anom}\}} \exp(\mathcal{M}^{(l)}_c(h,w)/\tau)}$$

Global and local gates modulate the mixtures to yield

$$\mathcal{P}_\mathrm{map} = \sum_{l=1}^L w_\mathrm{scale}[l] \left(M_\mathrm{spatial}^{(l)} \odot \mathcal{P}_\mathrm{anom}^{(l)}\right)$$

with the scalar anomaly score synthesized as $$S_\mathrm{final} = (1-\gamma) S_\mathrm{global} + \gamma S_\mathrm{local}$$ (Fu et al., 14 Jan 2026).

• Pixel-level Cosine Alignment (AHD): In video anomaly detection, each class $c$ (normal, abnormal) is represented by prompt embeddings. Multi-layer visual features are MLP-projected and cosine-aligned to each class mean. A 2-way softmax provides a probabilistic anomaly map (Gu et al., 1 Nov 2025).

3. Module Integration and Downstream Workflows

VTAM is typically not a monolithic network but an orchestrated composite of modules:

• Gallery Pruning and Feature Selection: In person anomaly retrieval, cosine similarities from the LHP are used to identify the top-$k$ nearest gallery candidates per text query. Only these are then cross-encoded and re-scored for final retrieval (Nguyen et al., 27 Nov 2025).
• Iterative Ensemble Re-Ranking: Instead of naive score averaging, successive model outputs are aggregated using an iterative, recall-at-1-optimized weighting at each step:

1. Start from $S \leftarrow 0$.
2. For each model, blend its scores into $S$ using the weight $w^*$ that maximizes recall@1.
3. Repeat for all ensemble members.

This approach allows high-confidence agreements to be locked in progressively, while ambiguous regions remain sensitive to subsequent models (Nguyen et al., 27 Nov 2025).

• Region-Aware Prompt Construction: For video anomaly detection, the RAE distills motion-differenced anomaly heatmaps into prompt vectors (region-localized, globally pooled, and learnable base prompts), concatenated as additional prefix tokens for a vision-language decoder, thus structuring where and when detected anomalies appear (Gu et al., 1 Nov 2025).

4. Quantitative Impact and Empirical Results

Empirical evaluations of VTAM-facilitated frameworks set new benchmarks across several domains:

Task	Model	Main Metric	Score	Relative SOTA Gain
Text-based person anomaly retrieval	VTAM (1M split) (Nguyen et al., 27 Nov 2025)	R@1	89.23	+9.70 (vs. prior)
Zero-shot industrial AD (MVTec-AD)	SSVP+VTAM (Fu et al., 14 Jan 2026)	Image-AUROC	93.0%	+0.4% (vs. ablation)
Fine-grained video anomaly (UBnormal)	T-VAD+VTAM (Gu et al., 1 Nov 2025)	micro-AUC	94.8	Substantially higher

Key ablation studies demonstrate additive improvement from each VTAM component. For instance, in (Nguyen et al., 27 Nov 2025), adding LHP, UIT, and feature selection incrementally raised R@1 from 85.24 up to 88.37, and the ensemble phase to 89.23. In (Fu et al., 14 Jan 2026), inclusion of VTAM provided a measurable +0.4% Image-AUROC and +0.1% Pixel-AUROC over HSVS+VCPG alone.

5. Design Significance and Comparative Position

VTAM's core significance lies in its strategies for fusing fine-grained spatial structure with high-level semantic alignment under visual-textual supervision, directly addressing the common challenge of balancing global semantic coverage with local discriminability. Key methodological contributions include:

• Local-global hybridization without incurring the limitations of single-scale encoders.
• Mixture-of-experts aggregation of multi-scale visual cues guided by text, using soft-attention/gating to dynamically adapt to anomaly characteristics.
• Pipeline integration (pruning, region reasoning, iterative ensembles) for both computational efficiency and representational precision.
• Capability for both dense pixel-level localization (heatmaps) and high-level anomaly reasoning via text-conditioned prompts.

Contrasted with conventional approaches that rely either on global or local-only backbones, or simplistic score pooling, VTAM-guided frameworks uniformly deliver superior sensitivity to subtle and context-dependent anomalies.

6. Associated Datasets and Evaluation Protocols

VTAM-integrating systems have been evaluated on:

• Pedestrian Anomaly Behavior (PAB): Evaluated for text-based person anomaly retrieval. Metrics: Recall@K (R@1, R@5, R@10) (Nguyen et al., 27 Nov 2025).
• MVTec-AD and other industrial inspection sets: SSVP+VTAM achieves state-of-the-art Image- and Pixel-AUROC (Fu et al., 14 Jan 2026).
• UBnormal and ShanghaiTech: Used for video anomaly detection and text-based Q&A. Metrics include micro/macro-AUC, RBDC, TBDC, BLEU-4, and Yes/No accuracy (Gu et al., 1 Nov 2025).

Evaluation protocols pair both detection/localization accuracy and cross-modal response fidelity, with ablations confirming each submodule's necessity for optimal performance.

7. Connections, Extensions, and Research Directions

A cross-cutting feature of VTAM is its extensible design: multi-scale feature integration, dual-gated calibration, and prompt-based region encoding are modular and can be adapted to diverse vision-language tasks. The consistent empirical benefit of VTAM across both retrieval and zero-shot anomaly detection contexts suggests a broader applicability to areas such as open-vocabulary segmentation, multimodal understanding, and explainable AI.

An ongoing research question is how to further optimize the interplay between local and global gates, as well as how the prompt construction logic in VTAM modules can best support downstream reasoning tasks. A plausible implication is that combining pixel-level alignment modules with language-driven structured prompt injection creates pipelines capable of more human-interpretable, interactive anomaly detection and explanation.

---

References: (Nguyen et al., 27 Nov 2025, Fu et al., 14 Jan 2026, Gu et al., 1 Nov 2025)