Sample-wise Label Fusion

• Sample-wise Label Fusion is a method that adapts fusion strategies to individual sample characteristics by combining diverse label signals.
• It employs techniques like vector concatenation, multi-annotator confusion matrices, non-local fusion, and rank-based retrieval to produce sample-specific inferences.
• Applications span medical imaging, multi-label classification, and ensemble methods, offering enhanced accuracy, calibration, and robustness over global approaches.

Sample-wise label fusion refers to the family of algorithmic strategies that combine multiple sources of label information—annotations, predictions, or per-class scores—for each data sample independently to produce a unified, sample-specific target or inference. Unlike global or static fusion, sample-wise approaches adapt fusion logic to the characteristics or uncertainties of each sample, and are highly relevant in multi-annotator supervision, ensemble classification, semi-supervised settings, medical imaging, and retrieval-based tasks.

1. Fundamental Architectures and Algorithms

Sample-wise label fusion can be instantiated through diverse methods depending on the problem domain and the sources of label signals:

• Vector Concatenation Models: LabelFusion’s AutoFusionClassifier fuses a traditional transformer embedding $h^{(b)}\in \mathbb{R}^d$ with a LLM–derived per-class score vector $s^{(\text{LLM})}\in\mathbb{R}^C$ for each text input, forming a sample-specific $$h^{(\text{fusion})} = [ h^{(b)} ; s^{(\text{LLM})} ] \in \mathbb{R}^{d+C}$$, which is then passed through a FusionMLP to output the final prediction (Schlee et al., 11 Dec 2025).
• Multi-annotator Confusion/Competence Approaches: Methods such as STAPLE consensus, average fusion, and random sampling operate per sample (medical image, instance) to aggregate multiple rater masks into a fused segmentation, retaining inter-rater uncertainty at the sample level (Lemay et al., 2022). Probabilistic fusion with sample-wise confusion matrices is prominent in multi-annotator settings; each sample $n$ is associated with one confusion matrix per annotator $P_n^{(r)}$ and a learned fusion vector $w_n$, yielding a soft target $y_n^{\text{targ}}$ specific to that sample (Gao et al., 2022).
• Non-local Atlas/Feature Fusion: In anatomical segmentation, CompareNet combines a voxel-wise classification unary score with a deep feature-based similarity term computed in a non-local window around each target voxel. The fusion is carried out individually at the sample (volume) and voxel levels (Liang et al., 2019).
• Rank-Based Retrieval Fusion: In extreme multi-label text classification (XMTC), sample-wise fusion involves aggregating candidate label rankings from separate retrieval systems (sparse BM25, dense BERT) per sample, using schemes such as CombSUM, CombMNZ, or Reciprocal Rank Fusion, followed by per-sample normalization and thresholding (França et al., 4 Jul 2025).
• Label-Wise Encoding and Fusion: LW-PT designs a per-label encoder and for each document stacks the outputs across all labels (regardless of ground-truth presence) into a single sample-specific representation, which is passed as input to the multi-label classifier (Liu et al., 2020).

2. Mathematical Foundations of Sample-wise Fusion

Key mathematical structures underlying sample-wise label fusion include:

• Concatenation and MLP Fusion: Given $h^{(b)}$ and $s^{(\text{LLM})}$, the AutoFusionClassifier forms $$h^{(\text{fusion})} = [ h^{(b)} ; s^{(\text{LLM})} ]$$, followed by $h_1 = \text{ReLU}( W_1 h^{(\text{fusion})} + b_1 )$, $z = W_2 h_1 + b_2$, and probabilities via softmax or sigmoid (Schlee et al., 11 Dec 2025).
• Sample-wise Confusion Matrices: In multi-annotator settings, for sample $n$, annotator $r$ has a confusion matrix $P_n^{(r)} \in \mathbb{R}^{K \times K}$; the cleaned label is $y_n^{(r,cln)} = P_n^{(r)} y_n^{(r)}$ (with $y_n^{(r)}$ one-hot), and the final soft target is $$y_n^{\text{targ}} = \sum_{r=1}^{R} w_{n,r} y_n^{(r,cln)}$$ (Gao et al., 2022).
• STAPLE EM Consensus: For segmentation, the posterior of the true label per voxel $i$ is $q_i$, computed via EM using all rater masks and rater-specific sensitivity/specificity, yielding $q_i = P(t_i=1|y_i^{(1:R)})$ (Lemay et al., 2022).
• Deep Non-local Fusion: At each target voxel $p$, CompareNet computes $$S_{p}(T) = S_{\text{unary},p}(T;\theta_{u}) + \alpha S_{\text{pairwise},p}(T;\theta_{p})$$, with $S_{\text{pairwise},p}(T)$ a weighted sum over nearby atlas voxels using learned similarity (Liang et al., 2019).
• Rank-based Fusion Algorithms: Typical formulas are:
  • $$F_{\text{CombSUM}}(s,\ell) = f'_s(s,\ell) + f'_d(s,\ell)$$
  • $$F_{\text{CombMNZ}}(s,\ell) = (f'_s(s,\ell) + f'_d(s,\ell)) \cdot c(s,\ell)$$
  • $$F_{\text{RRF}}(s,\ell) = 1/(k+r_s(s,\ell)) + 1/(k+r_d(s,\ell))$$ (França et al., 4 Jul 2025).

3. Application Domains and Empirical Performance

Sample-wise label fusion is employed across varied domains:

• Medical Imaging (Inter-Rater and Atlas Fusion): STAPLE, average fusion, and random sampling all provide sample-specific fusion masks for segmentation; SoftSeg regression frameworks result in superior calibration and preservation of inter-rater uncertainty, with ECE reduced to ≈2–3% and MAE on predictive entropy cut by 45–50% (Lemay et al., 2022). CompareNet’s end-to-end deep sample-wise fusion achieves Dice scores surpassing classical methods (e.g., 84.5% vs. 80.2% on IBSR, 74.6% vs. 71.9% on MICCAI 2012) while offering enhanced robustness to pathologies (Liang et al., 2019).
• Multi-annotator Classification: Sample-wise confusion/fusion models outperform global approaches on MNIST, CIFAR-100, and ImageNet-100 (e.g., 92.49% vs. 87.72% accuracy on MNIST; up to 19.5 points improvement on ImageNet-100), with marked gains on samples subject to annotator-specific reliability shifts (Gao et al., 2022).
• Text Classification Ensembles: LabelFusion’s concatenation of transformer and LLM signals at the sample level yields 92.4% accuracy on AG News and enables cost-aware gating of LLM queries (Schlee et al., 11 Dec 2025). LW-PT’s sample-wise fusion via concatenation of label-specific encodings boosts Macro-F1 by up to +14 points over state-of-the-art baselines (Liu et al., 2020).
• Extreme Multi-label Retrieval: Sample-wise fusion of sparse and dense retrieval outputs improves nDCG@1, Precision@5 for both head and tail labels, with up to +2.5 absolute points gain on tail categories across benchmarks; CombMNZ over ZMUV-normalized scores consistently delivers highest coverage (França et al., 4 Jul 2025).
• Pairwise Multi-label Classifiers: Local fuzzy confusion-matrix–based sample-wise fusion corrects supports per test sample, with classifier weights based on per-sample mutual information; empirical results confirm enhanced macro-F1 and exact-match accuracy on imbalanced and dense multi-label datasets (Trajdos et al., 2017).

4. Calibration, Uncertainty, and Adaptivity

Sample-wise fusion methods are characterized by their ability to calibrate uncertainty and adapt fusion weights or mechanisms on a per-sample basis:

• Preservation of Inter-Rater Uncertainty: SoftSeg frameworks preserve the entropy of expert disagreement and output better-calibrated probabilities in medical segmentation. STAPLE and random-sampling fusion under SoftSeg minimize ECE and entropy-MAE, while average fusion yields greater underconfidence (Lemay et al., 2022).
• Per-sample Reliability Estimation: Sample-wise confusion matrices ($P_n^{(r)}$) and fusion weights ($w_n$) enable per-instance reliability modeling, capturing annotator–sample dependencies unaddressed by global models. The weighting further addresses bias in samples with annotator-dependent error structure (Gao et al., 2022).
• Information-theoretic Local Weighting: Local fuzzy confusion matrix models assess classifier competence via sample-specific mutual information, enabling dynamic weighting at inference for improved coverage of rare or ambiguous labels (Trajdos et al., 2017).
• Cost-aware Adaptivity: LabelFusion employs confidence gating, adaptive LLM budgeting, and disk caching—all sample-dependent—to optimize accuracy–cost trade-offs (Schlee et al., 11 Dec 2025).

5. Comparative Analysis with Global Fusion Schemes

Sample-wise fusion offers several distinct advantages over traditional global schemes:

• Higher empirical accuracy and coverage: Per-sample fusion adapts to sample-specific uncertainty, local structure, and annotator reliability, consistently outperforming majority voting, global confusion matrix, or single-model approaches (Gao et al., 2022, Lemay et al., 2022, Liu et al., 2020).
• Superior calibration characteristics: Regression-style sample-wise fusion is systematically less overconfident, achieves lower ECE, and matches inter-rater entropy more closely (Lemay et al., 2022).
• Robustness to heterogeneity: Sample-by-sample adjustment (as in local confusion matrix models or deep non-local fusion) allows the model to manage label/annotation density, dataset imbalance, and rare label performance without hand-tuned class-balancing (Trajdos et al., 2017, Liang et al., 2019).
• Efficient signal integration: Architectural approaches such as direct concatenation of expert scores or label-wise encodings support optimization via single backpropagation passes and permit flexible inclusion of variable sources or retrieval signals (Schlee et al., 11 Dec 2025, Liu et al., 2020, França et al., 4 Jul 2025).

6. Limitations, Extensions, and Future Directions

While sample-wise label fusion demonstrates broad empirical and algorithmic superiority, certain practical considerations and open research questions remain:

• Computational cost: Fine-grained sample-wise confusion modeling (e.g., with permutation matrix decompositions) incurs higher memory and computation cost per sample, although decomposition and batching strategies mitigate overhead (Gao et al., 2022).
• Task dependence: The optimal fusion strategy (STAPLE, average, random-sampling) can be task- and dataset-dependent; careful benchmarking is necessary (Lemay et al., 2022).
• Annotation structure scaling: Extensions to large numbers of annotators ($R$), missing-label scenarios, or multi-source fusion are discussed as future work, with permutation-matrix decomposition and masking as promising directions (Gao et al., 2022).
• Interpretability: The adaptive and dynamic nature of sample-wise fusion models introduces complexity in interpretation of fusion weights/local confusion structure, motivating the need for diagnostic metrics and visualization tools.
• Generalization: While label-wise fusion architectures permit flexible stacking (e.g., in LW-PT), concatenation-based models may benefit from attention/gating extensions for large-scale, highly-correlated label spaces (Liu et al., 2020).

Sample-wise label fusion thus represents a versatile paradigm for integrating heterogeneous, uncertain, or multi-source label signals per sample, enabling enhanced accuracy, calibration, uncertainty quantification, and robustness relative to global or static fusion approaches. Its continued evolution addresses core challenges in multi-label learning, ensemble prediction, annotation reliability modeling, and cost-effectiveness of inference.