Mutual Scoring Mechanism (MSM) in Anomaly Detection

• MSM is a zero-shot, training-free framework for anomaly detection and segmentation that exploits self-similarity among normal patches.
• It computes per-patch mutual scores by comparing Euclidean distances across unlabeled test samples using interval averaging.
• Empirical results on datasets like MVTec 3D-AD demonstrate significant improvements in both classification and segmentation performance.

A Mutual Scoring Mechanism (MSM) is a zero-shot, training-free framework for anomaly classification (AC) and anomaly segmentation (AS) in industrial visual inspection, leveraging unlabeled data by exploiting statistical self-similarity among normal patches across samples and the diversity of anomalous patches. MSM assigns each test-region (e.g., a patch in an image or a local 3D window in a point cloud) a score by comparing it to all other unlabeled test samples in the same modality, exploiting the domain property that normal patches are highly redundant, while anomalies are typically isolated. MSM is the central component of the MuSc (Li et al., 2024) and MuSc-V2 (Li et al., 13 Nov 2025) zero-shot anomaly detection frameworks.

1. Formal Definition and Conceptual Foundation

In MSM, for a set of $N$ unlabeled test samples $\{I_1, ..., I_N\}$ (images or point clouds), each sample is divided into $M$ patch tokens via a feature backbone (e.g., Vision Transformer for images, Point-Transformer for point clouds). For every patch $m$ in sample $i$, a mutual score is computed by comparing its feature to the patches in all other samples $j \ne i$. The central intuition is that normal patches find many close patch-level neighbors across other test samples, while anomalous ones are distinct, resulting in higher mutual distances.

The per-patch mutual score is defined as:

$$a^{i,s,m}(j) = \min_{n} \left\| F^{i,s}(m) - F^{j,s}(n) \right\|_2,$$

where $F^{i,s}(m)$ is the feature of patch $m$ of sample $i$ at feature stage $s$, and the minimum is over patch indices $n$ in sample $j$.

Collecting all such per-sample comparisons $$A^{i,s,m} = \{ a^{i,s,m}(j) | j \in [1,N] \setminus \{i\} \}$$, MSM applies a normalization via interval averaging (IA), suppressing outliers and yielding a robust anomaly score for each patch.

2. Algorithmic Structure and Mathematical Steps

The practical implementation of MSM encompasses several stages:

1. Feature Preparation: Extract multi-scale patch (image) or local window (point cloud) features using neighborhood aggregation, e.g., Similarity Neighborhood Aggregation with Multi-Degrees (SNAMD) (Li et al., 13 Nov 2025) or Local Neighborhood Aggregation with Multiple Degrees (LNAMD) (Li et al., 2024). Each feature set $F^{i,s}$ forms the basis for mutual scoring.
2. Mutual Scoring: For every patch $m$ of $I_i$, its MSM score with respect to $I_j$ is the minimum $L_2$ (Euclidean) distance to patches in $I_j$:

$$a^{i,s,m}(j) = \min_{n} \| F^{i,s}(m) - F^{j,s}(n) \|_2.$$

These scores, across all $j \neq i$, form $A^{i,s,m}$.

1. Interval Averaging (IA): Sort $A^{i,s,m}$ and average the lowest $X\%$ to suppress noise:

$$\overline{a}^{i,s,m} = \frac{1}{K} \sum_{k=1}^{K} a^{i,s,m}(\overline{I}_k),$$

where $\overline{I}_1, ..., \overline{I}_K$ correspond to the $K = \lceil X\% \cdot (N-1) \rceil$ lowest-scoring samples.

2. Multi-Stage Fusion: Aggregate the IA-normalized scores across feature stages $s = 1,...,S$ by averaging:

$$a^{i,m} = \frac{1}{S} \sum_{s=1}^S \overline{a}^{i,s,m}.$$

3. Anomaly Map & Sample Classification: Reassemble $\{a^{i,m}\}$ into the spatial layout for segmentation; for AC, use $c_i = \max_m a^{i,m}$ as the sample-level score.

In multimodal settings (e.g., MuSc-V2), Cross-modal Anomaly Enhancement (CAE) aligns, rescales, and merges 2D and 3D anomaly maps, applying variance-based confidence weights to regulate cross-modal contributions.

3. Mutual Scoring within Modality and Neighborhood Structures

Unlike classical anomaly detection methods that reference a "normal" bank or require labels, MSM compares each patch against all other unlabeled samples. The per-sample minimum neighbor distance is statistically discriminative: for normal patches, the minimum distance is typically small due to many similar regions elsewhere; for anomalies, distances stay high owing to their uniqueness.

By collecting a vector of such distances for each patch and leveraging interval average normalization, MSM counteracts atypical background variations and misalignment artifacts, further reinforcing its zero-shot robustness. This mechanism applies independently in either 2D or 3D feature space, supporting single-modality or multimodal fusion.

4. Score Fusion, Normalization, and Multimodal Extensions

MSM normalizes scores through two primary mechanisms:

• Interval Average (IA): IA reduces the influence of rare outlier matches by averaging only the lowest $X\%$ rather than all patch-to-image distances, improving anomaly/normal separation.
• Stage Fusion: Averaging across multiple feature scales increases robustness to defect size and localization variations.

In the multimodal case, CAE aligns and fuses 2D and 3D anomaly maps by,

$$a_I \leftarrow a_I + \lambda \max(a_{P \to I}, a_I),$$

where the confidence weight $\lambda = 1-\mathrm{std}(A_{P \to I})$ down-weights noisy cross-modal projections. Stage-, modality-, and neighborhood-wise aggregation ensures the resulting anomaly maps capture various spatial extents and suppress false positives.

5. Integration with Auxiliary Framework Components

MSM operates as the central computation engine but relies on complementary methods:

• SNAMD/LNAMD: Preprocessing of patch features via neighborhood- and degree-aware aggregation (multi-scale pooling, similarity weighting) improves feature discrimination and context sensitivity.
• CAE: In multi-sensor settings, CAE post-processes MSM outputs for each modality—importantly, allowing 3D cues to recover weak or missing anomalies in 2D (and vice versa).
• RsCon/RsCIN: Re-scoring with Constrained Neighborhoods refines sample-level anomaly scores by constructing a $k$-NN graph on anomaly-salient features and mixing $c_i$ with its neighbors to suppress outliers due to extremely weak or noisy signals.

The complete operational pipeline ensures adaptability and robustness across datasets of varying sizes and modalities, evidenced by the modular structure adopted in both MuSc (Li et al., 2024) and MuSc-V2 (Li et al., 13 Nov 2025).

6. Ablation Insights and Empirical Performance

Ablation studies in MuSc-V2 show that MSM and its interval averaging are pivotal for state-of-the-art results. For example, on MVTec 3D-AD, disabling IA decreases F1-max_cl by 0.6 points (93.0→92.4) and AP_seg by 2.2 points (54.7→52.5); removing CAE further reduces F1-max_cl by 1.3 points (93.0→91.7) and AP_seg by 2.6 points. Omitting the variance-based $\lambda$ penalty yields further drops (F1-max_seg -0.2, AP_seg -0.4). In MSM’s ablations on MuSc (Li et al., 2024), using IA with 30% minimum-distance improves image-AUROC by +2.8% and pixel-AUROC by +0.5% over mean-aggregation (Tab. 6), demonstrating the necessity of interval averaging for robust performance.

Empirical results show MSM-based zero-shot models surpass prior training-based and few-shot methods for both AC and AS. MuSc-V2 attains a $+23.7\%$ AP-segmentation gain over prior zero-shot baselines on MVTec 3D-AD and a $+19.3\%$ gain on Eyecandies. Performance remains robust even under large subset splitting ($g=2$, $g=3$), with drops of at most 1%, and when normal/anomaly mix varies down to 0% normal, with segmentation AUROC dropping by less than 0.3%.

7. Implementation Considerations and Scalability

MSM is computationally intensive, with naive complexity $O(N^2 M^2 |R| L)$, but can be parallelized and subdivided. Precomputing features, using efficient nearest-neighbor search (e.g., approximate k-NN with Faiss), and applying subset splitting ($s=2$ or $s=3$) yield practical inference times under 1 s/image for $N\lesssim200$, $M\approx2300$, $|R|\cdot L=12$ on a single RTX3090 (Li et al., 2024).

Key hyperparameters include:

• Aggregation degrees $R=\{1,3,5\}$ (multi-scale spatial pooling).
• Interval average percentage $X=30\%$ (distance normalization).
• Neighborhood sizes for RsCIN/RsCon ($k_1$, $k_2$). Empirical tuning demonstrates stability across these parameters. All normalization operations are vectorized and amenable to distributed computation.

Summary Table: Mutual Scoring Mechanism in MuSc-V2 and MuSc

Component	Purpose	Empirical Effect (Ablation)
Interval Average (IA)	Outlier suppression	−0.6 to −2.2 points if ablated (Li et al., 13 Nov 2025)
CAE (cross-modal)	Modality fusion/recovery	−1.3 to −2.6 points if ablated
Confidence weight λ	Down-weighting variance	−0.2 to −0.4 points if ablated

Each component is substantiated as essential for robust zero-shot anomaly detection and segmentation in industrial scenarios, underpinned by the consistent gains observed in empirical benchmarks.

---

References:

• "MuSc: Zero-Shot Industrial Anomaly Classification and Segmentation with Mutual Scoring of the Unlabeled Images" (Li et al., 2024)
• "MuSc-V2: Zero-Shot Multimodal Industrial Anomaly Classification and Segmentation with Mutual Scoring of Unlabeled Samples" (Li et al., 13 Nov 2025)