Pseudo-Label Loss

• Pseudo-Label Loss is a semi-supervised objective that uses automatically generated labels from clustering, teacher–student models, and heuristics to train models without full supervision.
• It encompasses hard, soft, weighted, and contrastive loss formulations to effectively address label noise, sample imbalance, and error propagation.
• Refinement techniques like dynamic thresholding and confidence weighting enhance its robustness and improve performance in tasks such as unsupervised domain adaptation and segmentation.

A pseudo-label loss is a supervised or semi-supervised objective defined over automatically generated labels (“pseudo-labels”) assigned to unlabeled or weakly annotated data during training. These pseudo-labels typically originate from clustering, teacher–student models, heuristic rules, or external models, and may be “hard” (one-hot) or “soft” (probabilistic/distributional). The pseudo-label loss usually supplements or replaces standard loss terms in settings without complete supervision and is a central mechanism in unsupervised domain adaptation, semi-supervised classification, segmentation, and weakly supervised or partial-label learning. The definition, formulation, and strategic weighting of pseudo-label loss must address label noise, sample imbalance, error propagation, and optimization stability.

1. Formulation and Types of Pseudo-Label Loss

Pseudo-label loss functions commonly operate on model outputs evaluated against pseudo-label targets and may be designed as classification loss, triplet or metric loss, contrastive loss, regression loss (for pseudo-label scores), or reconstruction loss (autoencoders). They fall into several broad categories:

• Hard Pseudo-Label Loss: Supervision via one-hot labels, possibly from cluster assignments or teacher predictions, with standard cross-entropy or similar objectives (Ge et al., 2020, Scherer et al., 2022).
• Soft Pseudo-Label Loss: Supervisory targets are real-valued probability distributions or distributions over classes, requiring soft cross-entropy or KL-divergence style losses (Ge et al., 2020, Chen et al., 2024).
• Weighted Pseudo-Label Loss: Instances (or pixels) are weighted by model confidence, predicted mask IoU, or external energy scores to de-emphasize unreliable pseudo-labels (Scherer et al., 2022, Hu et al., 2023, Zhang et al., 2024).
• Contrastive Pseudo-Label Loss: In segmentation and representation learning, pseudo-labels define positive pairs (same class under pseudo-label) and negatives (different class) for use in local or global contrastive losses (Chaitanya et al., 2021).
• Pseudo-Label Correction and Dynamic Re-weighting: Multiple strategies, including label refinement, multi-focus or multi-round aggregation, and curriculum weighting, have been proposed to dynamically update or calibrate pseudo labels or to modulate their influence during training (He et al., 2023, Tran et al., 28 Aug 2025, Zhang et al., 2022, Zhang et al., 2024).

The following table summarizes typical pseudo-label loss formulations across key research lines:

Domain/method	Target (pseudo-label)	Loss type(s)
Clustering-based re-ID (Ge et al., 2020)	Hard (cluster ID), Soft (EMA)	Cross-entropy, triplet, soft triplet
Multi-label/SPML (Chen et al., 2024, Tran et al., 28 Aug 2025)	Soft probability estimate	Robust BCE/MAE hybrid
Segmentation (Scherer et al., 2022)	Hard (pixel-wise), confidence weighted	SCE, Dice, weighting by softmax confidence
Weakly supervised (LLP) (Ma et al., 2024)	Bag-averaged predictions, instance-level pseudo	Cross-entropy, adaptive entropy weights
Test-time adaptation (Han et al., 2023)	Complementary labels (not-in-class)	Filtered NLL/cross-entropy

2. Challenges Associated with Pseudo-Label Loss: Noise, Confirmation Bias, and Imbalance

Pseudo-label loss functions inherently suffer from the noise in the pseudo-label assignment process. This noise arises due to:

• Imperfect teacher/student model predictions or clustering (Ge et al., 2020, Tran et al., 28 Aug 2025).
• Domain shift, causing erroneous high-confidence pseudo-labeling (Han et al., 2023).
• Incomplete or missing label annotations, resulting in prevalence of false negatives (Chen et al., 2024, Zhang et al., 2022).
• Sample or class imbalance, especially in long-tailed scenarios, leading to underrepresentation of minority classes (Zhang et al., 2024).

Empirical findings highlight several key effects:

• Pseudo-label noise can severely degrade model performance, particularly when naively used with standard loss functions (Ge et al., 2020, Scherer et al., 2022).
• Confirmation bias emerges when models reinforce early, erroneous pseudo-labels, limiting the utility of future pseudo-label correction (Scherer et al., 2022, Zhang et al., 2024).
• Hard negative mining or thresholding can inadvertently ignore informative or challenging examples, especially for underrepresented classes (Hu et al., 2023, Tran et al., 28 Aug 2025, Zhang et al., 2024).

3. Refinement, Weighting, and Robustification Strategies

Efforts to mitigate pseudo-label loss drawbacks focus on dynamic refinement, robust losses, and weighting schemes:

• Temporal ensembling and mutual mean-teaching: Use exponential moving averages of network weights to generate more stable soft pseudo-labels as supervisory targets (Ge et al., 2020).
• Dynamic thresholding: Dynamically adapt selection thresholds for pseudo-label acceptance, either globally (via EMA) or per-class, balancing between coverage and reliability (Zhang et al., 2024, He et al., 2023).
• Confidence/energy-based weighting: Pseudo-label loss is multiplied by measures of model confidence (softmax, IoU, or energy scores), down-weighting likely erroneous pseudo-labels (Zhang et al., 2024, Scherer et al., 2022, Hu et al., 2023).
• Robust surrogate losses: Losses such as generalized cross-entropy (GCE), symmetric cross-entropy (SCE), and custom robust loss functions interpolate between cross-entropy and MAE to increase tolerance to label noise (Chen et al., 2024, Cui et al., 2022).
• Soft triplet and contrastive losses: Adapting metric-learning objectives to operate over soft distributions derived from pseudo-labels rather than hard assignments enables more robust feature discrimination under uncertainty (Ge et al., 2020, Chaitanya et al., 2021).
• Regularization and expectation alignment: Regularizers force the empirical number of positives predicted by the network to match population-level statistics, preventing model collapse under unreliable pseudo-label proportions (Tran et al., 28 Aug 2025, Zhang et al., 2022).

4. Integration with Broader Learning Frameworks and Objectives

Pseudo-label loss functions are integral to various learning paradigms and workflow architectures, including:

• Teacher–student/self-training and Mutual Teaching: Pseudo-labels are either generated by a static teacher or jointly refined between multiple networks with cross supervision (Ge et al., 2020, Cui et al., 2022).
• Bag-level and instance-level loss in LLP: Bag-level losses align average predictions with label proportions, while auxiliary instance-level pseudo-label loss (with confidence-adaptive weighting) improves representation learning (Ma et al., 2024).
• Vision-language and external pseudo-labeling: External models (e.g., CLIP) produce pseudo-labels which are dynamically updated (e.g., through DAMP) and robustly integrated via a specialized pseudo-label loss (Tran et al., 28 Aug 2025).
• Hybrid augmentation and mixing: Image or feature mixing (e.g., cow-pattern masks, patch-based, and multi-focus strategies) perturbs pseudo-label consistency, promoting greater robustness (Scherer et al., 2022, Hu et al., 2023, Tran et al., 28 Aug 2025).
• Complementary and energy-based pseudo-labeling: Instead of assigning “most likely” class, negative or “complementary” labels identify the set of classes an input is unlikely to belong to (Han et al., 2023); alternatively, energy functions filter in-distribution samples for pseudo-label selection (Zhang et al., 2024).

5. Empirical Impact and Comparative Results

Assessments using benchmark datasets consistently demonstrate the importance and impact of accurate pseudo-label supervision:

• Mutual mean-teaching and soft pseudo-label usage yield mAP improvements up to 18.2% in unsupervised person re-ID DA (Ge et al., 2020).
• Robust loss variants (GCE, BCE, SCE) substantially outperform standard CE when using noisy pseudo-labels for both classification and segmentation, reaching performance close to fully supervised settings (Cui et al., 2022).
• Instance-wise weighting, dynamic thresholding, and class-adaptive margins provide superior mean Average Precision or Dice scores in scenarios with partial labels, severe class imbalance, or large bag sizes (Chen et al., 2024, Zhang et al., 2022, Ma et al., 2024, Zhang et al., 2024).
• Vision-language and multi-view dynamic pseudo-labeling, coupled with robust GPR loss, set new benchmarks on SPML and generic multi-label classification datasets (Tran et al., 28 Aug 2025).
• Loss smoothing to remove derivative discontinuities in pseudo-label loss increases stability and improves error rates in low-label regimes (Karaliolios et al., 2024).

6. Open Issues and Future Directions

Early pseudo-label misassignments and lasting confirmation bias continue to present challenges, and performance can sometimes be non-monotonic as the labeled set grows (Karaliolios et al., 2024). Prospective research areas include:

• Incorporation of uncertainty and calibration mechanisms in pseudo-label assignment and loss scaling (Tran et al., 28 Aug 2025, Zhang et al., 2024).
• Adaptive curriculum learning and dynamic balancing between supervised and pseudo-label loss (Zhang et al., 2022, Tran et al., 28 Aug 2025).
• Expanding direct loss construction from heuristics or rule-based sources to circumvent or generalize pseudo-label aggregation (Sam et al., 2022).
• Hybrid strategies for open-set, out-of-distribution, and long-tailed recognition, leveraging dynamic label filtering and instance-aware losses (Zhang et al., 2024, Zhang et al., 2024).
• Imposing monotonicity guarantees (i.e., performance should not degrade with increasing labeled data) to reconcile semi-supervised and active learning pipelines (Karaliolios et al., 2024).
• Further exploring the integration of vision-language pseudo-labeling and spatial/patch-level aggregation for improved context exploitation (Tran et al., 28 Aug 2025).

7. Summary Table: Pseudo-Label Loss Strategies in Recent Literature

Paper / Domain	Pseudo-Label Source	Noise Handling Strategy	Key Performance Effect
(Ge et al., 2020) Person Re-ID	Clustering + EMA	Soft triplet loss, mutual teaching	+18.2% mAP DA gain
(Chen et al., 2024) SPML	Model probs, soft k(p)	Robust MAE/BCE loss; instance weight	Reduces false negatives, mAP↑
(Scherer et al., 2022) Segmentation	Teacher prediction	Symmetric CE; dynamic weight/filter	mIoU +13.5% Cityscapes
(Tran et al., 28 Aug 2025) SPML, multi-label	VLM (CLIP), DAMP	GPR Loss; dynamic multi-focus labels	SOTA mAP; robust to label noise
(Zhang et al., 2024) SAR ATR	Energy-based filter	Adaptive margin/triplet loss	+1.2pt onMSTAR IR30, robust minority
(Karaliolios et al., 2024) SSL classification	Model prediction	Smooth factor for loss continuity	<2.5% error vs FixMatch, ↑stability

• Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification (Ge et al., 2020)
• A Pseudo-labelling Auto-Encoder for unsupervised image classification (Bouayed et al., 2020)
• Adaptive Affinity Loss and Erroneous Pseudo-Label Refinement for Weakly Supervised Semantic Segmentation (Zhang et al., 2021)
• Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation (Chaitanya et al., 2021)
• PARS: Pseudo-Label Aware Robust Sample Selection for Learning with Noisy Labels (Goel et al., 2022)
• Semi-supervised Learning using Robust Loss (Cui et al., 2022)
• Pseudo-Label Noise Suppression Techniques for Semi-Supervised Semantic Segmentation (Scherer et al., 2022)
• An Effective Approach for Multi-label Classification with Missing Labels (Zhang et al., 2022)
• Losses over Labels: Weakly Supervised Learning via Direct Loss Construction (Sam et al., 2022)
• Rethinking Precision of Pseudo Label: Test-Time Adaptation via Complementary Learning (Han et al., 2023)
• Pseudo-label Correction and Learning For Semi-Supervised Object Detection (He et al., 2023)
• Pseudo-label Alignment for Semi-supervised Instance Segmentation (Hu et al., 2023)
• Multi-Task Pseudo-Label Learning for Non-Intrusive Speech Quality Assessment Model (Zezario et al., 2023)
• Boosting Single Positive Multi-label Classification with Generalized Robust Loss (Chen et al., 2024)
• Smooth Pseudo-Labeling (Karaliolios et al., 2024)
• Self Adaptive Threshold Pseudo-labeling and Unreliable Sample Contrastive Loss for Semi-supervised Image Classification (Zhang et al., 2024)
• Energy Score-based Pseudo-Label Filtering and Adaptive Loss for Imbalanced Semi-supervised SAR target recognition (Zhang et al., 2024)
• Forming Auxiliary High-confident Instance-level Loss to Promote Learning from Label Proportions (Ma et al., 2024)
• More Reliable Pseudo-labels, Better Performance: A Generalized Approach to Single Positive Multi-label Learning (Tran et al., 28 Aug 2025)
• Point2RBox-v3: Self-Bootstrapping from Point Annotations via Integrated Pseudo-Label Refinement and Utilization (Zhang et al., 30 Sep 2025)