Webly Supervised & Noisy Label Protocols

• Webly supervised learning is a scalable approach that gathers web data with automatically assigned, yet noisy, labels and faces challenges like in-distribution and out-of-distribution noise.
• Advanced protocols employ loss-based sample selection, dynamic pseudo-labeling, and meta-learning to correct label errors and improve model robustness.
• Emerging techniques integrate visual-semantic fusion, prototype-based correction, and curriculum strategies to effectively mitigate noise and open-set errors.

Webly supervised learning and noisy label protocols are foundational to the modern scaling of deep neural networks on massive, real-world datasets acquired from the web, search engines, or crowdsourcing platforms. These protocols contend with the fact that labels attached to such data are inherently noisy, often afflicted by semantic confusion, out-of-distribution (OOD) errors, and class imbalance. This article presents a comprehensive, technical overview of the methodological landscape that enables robust model training from webly sourced and noisy-labeled data.

1. Webly Supervised Learning: Data Sources, Noise Taxonomy, and Challenges

Webly supervised learning (WSL) relies on automatically collecting datasets by scraping the web, using queries, tags, or metadata as labels. This paradigm offers scalability and diversity but introduces complex label noise patterns. The principal types of noise encountered in WSL are:

• In-Distribution (ID) Noise: Label flips among the known target classes (e.g., 'cat' mislabeled as 'dog'), often modeled as a label transition process.
• Out-of-Distribution (OOD) Noise: Samples whose true labels are external to the target label set (e.g., a 'rabbit' labeled as ‘tiger’ when querying 'tiger'); OOD often dominates real web noise, with empirical rates as high as 24% OOD and 5% ID in mini-WebVision (Albert et al., 2021).
• Semantic/Clustered Noise: Query polysemy or tag ambiguity (e.g., “drumstick”: chicken legs, percussion mallet) producing visually coherent but semantically off-target clusters (Yang et al., 2020).
• Class Imbalance and Long-Tailed Distributions: Frequently, web crawls induce heavy-tailed class distributions, deviating from synthetic noise setups.

These real-world noise patterns are not well-captured by simple symmetric/asymmetric label-flip models, necessitating tailored protocol design (Ortego et al., 2019, Albert et al., 2021).

2. Sample Selection, Label Correction, and Meta-Learning Strategies

Robust learning under noisy webly labels typically involves sophisticated sample selection and pseudo-labeling protocols, often combined with meta-learning or semi-supervised learning (SSL) components.

2.1. Loss/Feature GMM Selection

Many protocols partition data into confident (clean) and ambiguous/noisy subsets using per-sample loss or feature statistics, often via Gaussian Mixture Models (GMMs) or Beta-Mixture Models (BMMs):

• Loss-small methods: Select clean samples based on the posterior probability of belonging to the low-loss component in a two-component GMM over per-sample cross-entropy losses (Wang et al., 2022, Bai et al., 2024, Albert et al., 2022).
• Feature-space clustering: Use feature similarity (e.g., cosine similarity to class centroids) and GMMs to delineate structural agreement (Bai et al., 2024).
• Combined selection: Parallel use of loss- and feature-based GMMs, followed by a meta-classifier ('meta sample purification') operating on the joint scores, as in Two-Stream Sample Distillation (TSSD) (Bai et al., 2024).

2.2. Pseudo-labeling and Dynamic Label Regression

Noisy samples not confidently classified as clean are often assigned pseudo-labels based on model predictions:

• Joint Optimization: Continuously refine soft pseudo-labels across epochs using the running model predictions, as in Distribution-Robust Pseudo-Labeling (DRPL) (Ortego et al., 2019).
• Dynamic relabeling: Pseudo-labels are updated per epoch, and samples are dynamically re-assigned to the clean/unlabeled partitions using thresholds from fitted GMMs/BMMs (Wang et al., 2022, Bai et al., 2024, Ortego et al., 2019).

2.3. Noise-Aware Meta-Learning

Meta-learning protocols utilize a small trusted set (if available) to estimate per-sample weights or correction parameters by optimizing for improved performance on the clean subset (Zhang et al., 2019). This entails bi-level optimization: first adapt model weights on noisy data (possibly using weights*/pseudo-labels*), then optimize sample weights/pseudo-label assignments to maximize clean validation accuracy.

3. Label Uncertainty, Dynamic Softening, and ID/OOD Discrimination

Recent protocols focus on explicitly distinguishing ID versus OOD noise and applying differentiated correction/softening strategies.

• DSOS (Dynamic Softening of OOD Samples) (Albert et al., 2021): Utilizes the collision entropy of interpolated labels

$$l_{\text{detect}} = -\log \sum_{c=1}^C \left(\frac{y_c + \hat{z}_c}{2}\right)^2$$

to fit a Beta-Mixture Model and obtain per-sample ID and OOD probabilities $(u_i, v_i)$. Clean/ID samples are corrected by bootstrapping with model predictions, OOD samples are pushed toward uniform targets with a softmax temperature, and an entropy regularization encourages uncertainty on OOD data.

• PLS (Pseudo-Loss Selection) (Albert et al., 2022): Computes a pseudo-loss (cross-entropy of the guessed pseudo-label against the model's unaugmented prediction), fitting a GMM to distinguish confident corrections, and applies confidence-guided weighting and interpolation between supervised and unsupervised contrastive losses.

These approaches substantially outperform earlier methods when OOD dominates, as is typical in web noise (Albert et al., 2021).

4. Visual-Semantic and Prototype-Based Correction

Protocols increasingly exploit side information—metadata, textual context, or visual neighborhood—to identify clean anchors and perform robust label correction.

• VSGraph-LC (Yang et al., 2020): Constructs a k-NN graph on visual features and smooths metadata across the graph; metadata-guided anchors supervise a GNN label propagation. The output pseudo-labels (combining GNN and CNN) are used to relabel the full web set, leading to robust open-set and in-distribution noise correction.
• MoPro/CAPro (Li et al., 2020, Qin et al., 2023): Leverage class prototypes (momentum-updated) in embedding space; each sample’s prediction is fused with its prototype-based similarity for label correction and OOD rejection. Cross-modality methods extend this to text/image prototype alignment, using graph convolution to enhance text features via visual neighbors, and prototype-regularized contrastive learning to enforce cluster structure.

Such algorithms excel at suppressing polysemous or densely clustered off-target samples characteristic of web noise and fine-grained domains.

5. Multi-Stream, Curriculum, and Semi-Supervised Paradigms

Effective protocols are often structured in multiple training stages or streams, each tuned for noise-robustness:

• Curriculum protocols (Chen et al., 2015): Train first on 'easy' sets (e.g., Google images) to build initial representations, then adapt to 'hard' (Flickr) data with noise-aware graph regularization; curriculum mitigates confirmation bias.
• Co-learning frameworks (Tan et al., 2021): Couple supervised and self-supervised branches with shared feature encoders, regularizing via agreement terms; the self-supervised branch helps to decorrelate from noisy labels, and co-regularization aligns structure in both spaces.
• Two-stage/semi-supervised learning (Ding et al., 2018, Bai et al., 2024, Ortego et al., 2019, Wang et al., 2022): Identify a trusted labeled subset, treat the remainder as unlabeled, and apply SSL mechanisms (e.g., MixMatch, consistency regularization) for further training.

A summary of predominant protocol axes is depicted in the table below:

Protocol	Selection/Correction Mechanism	Noise Model	Open/OOD Support
TSSD (Bai et al., 2024)	Two-stream PSD + MSP, meta-classifier	GMM (loss/feature)	Partial
DSOS (Albert et al., 2021)	Dynamic bootstrapping/softening	BMM collision entropy	Explicit
MoPro (Li et al., 2020)	Momentum prototypes, contrastive loss	Implicit (no T needed)	Explicit
VSGraph-LC (Yang et al., 2020)	Visual-semantic k-NN graph + GNN	Metadata/graph-based	Yes (open set F1)
CAPro (Qin et al., 2023)	Textual/visual prototypes + bootstrapping	Polysemy + OOD (joint)	Yes (open set F1)
PLS (Albert et al., 2022)	Pseudo-loss GMM, weighted objectives	Dual-stage GMM	Yes
LCCN (Yao et al., 2019)	Dynamic label regression (Dirichlet prior)	Class-cond. transition	Yes (K+1 output)

6. Practical Guidelines and Future Directions

Key empirical and procedural recommendations include:

• Augmentation: Hybrid weak/strong (e.g., MixUp, RandAugment) schemes help delay overfitting and provide label correction signals (Wang et al., 2022).
• Class-balancing: Uniform regularizers or stable prototype momentum improve performance on long-tailed web datasets (Li et al., 2020, Wang et al., 2022).
• Hyperparameter robustness: Methods such as VSGraph-LC, MoPro, CAPro, and DSOS show broad insensitivity to exact GMM/BMM thresholds, mixup/contrastive weights, or k-NN parameters (Yang et al., 2020, Li et al., 2020, Qin et al., 2023, Albert et al., 2021).
• Open-set detection: Prototype and entropy-based strategies (e.g., DSOS, CAPro) are essential to reliably reject OOD noise that comprises a significant portion of web supervision (Albert et al., 2021, Qin et al., 2023).
• Metadata and cross-modal fusion: Text embeddings, graph smoothing, and cross-modality alignment are increasingly central for noise disambiguation in real-world WSL scenarios (Yang et al., 2020, Qin et al., 2023, Yang et al., 2020).

Methodological innovation continues, with ongoing development in end-to-end refined graph-label pipelines (Yang et al., 2020), confidence-aware bootstrapping (Qin et al., 2023), fully automatic metadata-guided anchor discovery (Yang et al., 2020, Qin et al., 2023), and robust single-network loss design (e.g., DSOS (Albert et al., 2021)). There is an ongoing shift toward frameworks that do not require a small clean set (self-contained confidence, graph-based correction), and to those that unify robust representation learning with explicit open-set and semantic noise mitigation.

7. Benchmark Datasets and Standard Evaluation Protocols

Empirical validation relies on both real-world, webly-crawled datasets and controlled synthetic benchmarks:

• WebVision-1000/V1.0: 2.44 M images (∼34% OOD, ∼16% class noise), ILSVRC label overlap.
• NUS-81-Web / NUS-WIDE (Web): Flickr-sourced, ∼50% tag-level FP noise, 81/81 classes; strong test-bed for multi-label and open-set tasks.
• Clothing1M: ∼1 M e-commerce images, 14 categories, ∼39% real label noise.
• Food-101N, Animal-10N, MiniImageNet: Used for fine-grained webly-labeled robustness tests.

Metrics are typically top-1/top-5 error, macro-F1, mean AP, and open-class F1 (for OOD support), all quantified on held-out, manually curated test splits.

---

In sum, the spectrum of protocols for webly supervised and noisy-label learning—spanning GMM/BMM selection, pseudo-labeling, graph-based label smoothing, prototype-driven alignment, and contrastive bootstrapping—constitutes a robust toolkit for harnessing the scale of web data in deep learning, with increasing technical sophistication in distinguishing, correcting, or suppressing the idiosyncratic label noise endemic to such sources (Yang et al., 2020, Bai et al., 2024, Wang et al., 2022, Albert et al., 2021, Qin et al., 2023).