Self-Supervised Speaker Embeddings

• Self-supervised speaker embeddings are fixed-dimensional representations learned from unlabeled speech using contrastive, clustering, and reconstruction methods, capturing speaker identity robustly.
• Leveraging iterative pseudo-labeling and heavy data augmentation, these methods effectively reduce channel bias and improve key metrics such as EER in speaker verification and diarization tasks.
• Advanced frameworks integrate hybrid loss functions and multimodal cues to narrow the performance gap with supervised systems, ensuring resilient and domain-robust speaker profiling.

Self-supervised speaker embeddings are fixed-dimensional vector representations of short or long speech segments learned without access to human-provided speaker labels. The objective is to induce embeddings that are maximally discriminative of speaker identity, robust to channel and content variation, and suitable for downstream speaker verification, diarization, or profiling—leveraging large unlabeled speech corpora. Modern self-supervised paradigms exploit contrastive, mutual-information, reconstruction, or clustering-based bootstrapping mechanisms, often in iterative or hybrid frameworks. These approaches have recently narrowed or eliminated the performance gap with fully supervised speaker encoding on verification and diarization benchmarks.

1. Core Self-Supervised Paradigms

The principal self-supervised paradigms for speaker embedding training are:

• Contrastive learning: Models such as SimCLR, MoCo, and InfoNCE jointly attract embeddings of positive (same-utterance, highly-augmented) segment pairs and repel negatives (other utterances). Formally, the InfoNCE loss for anchor $\mathbf{z}_i$ and positive $\mathbf{z}_i'$ in a batch is

$$\mathcal{L}_{\mathrm{InfoNCE}} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(\mathbf{z}_i \cdot \mathbf{z}_i' / \tau)}{\sum_{j=1}^N \exp(\mathbf{z}_i \cdot \mathbf{z}_j / \tau)}$$

(Lepage et al., 2022, Zhang et al., 2022, Xia et al., 2020).

• Non-contrastive learning: BYOL and DINO-style frameworks propagate representations through "online" and "target" encoders, matching their normalized predictions. The target is momentum-updated; loss is typically $\| \overline{p}_\theta - \overline{z}'_\xi \|_2^2$. These methods avoid explicit negatives (Klapsas et al., 2022, Zhang et al., 2022).
• Iterative clustering and bootstrapping: Embedding networks are iteratively trained with pseudo-labels from clustering their own outputs (e.g., k-means, AHC), refining speaker classes and embeddings (Cai et al., 2020, Singh et al., 2020). Prototypical and memory-bank extensions mitigate "class collision."
• Reconstruction with auxiliary information: Encoders are trained to reconstruct masked or future frames, or to synthesize other segments’ features conditioned on phone content, inducing the embedding to encode speaker-invariant information (Stafylakis et al., 2019, Baali et al., 20 Oct 2025).
• Adversarial invariance: Encoder is penalized if a small discriminator can infer augmentation channel, enforcing channel invariance in learned speaker representations (Huh et al., 2020).
• Information maximization/regulation: Losses such as Barlow Twins or VICReg enforce invariance, diversity, and non-collapsing codes (Lepage et al., 2022).

Historically, unsupervised i-vector modeling via total variability was replaced by deep x-vector and ResNet/TDNN architectures. The inability of softmax-trained x-vectors to leverage unlabeled data motivated these SSL approaches (Stafylakis et al., 2019, Zhang et al., 2022).

2. Neural Architectures and Data Augmentation

Speaker embedding extractors in self-supervised settings universally adopt neural architectures designed for robust temporal aggregation:

Architecture	Input features	Pooling method	Embedding dim	Reference
ResNet-34/SE34L	log-Mel, MFCC, waveform	Attentive/Statistics	512–2048	(Cai et al., 2020, Zhang et al., 2022)
ECAPA-TDNN	log-Mel, MFCC	Channel attn. + SAP	512–2048	(Zhang et al., 2022, Miara et al., 2024)
Fast ResNet-34	log-Mel	SAP	512	(Huh et al., 2020)
TDNN/Kaldi x-vector	MFCC/log-Mel	Stats pooling	128–1024	(Stafylakis et al., 2019, Xia et al., 2020)
DELULU	raw waveform	Mean-pool (after Transformer)	512+	(Baali et al., 20 Oct 2025)

Heavy augmentation is systematically used: additive noise/music/babble (MUSAN), simulated RIR-based reverberation (Xia et al., 2020, Huh et al., 2020), log-mel domain manipulations (mixup, resize-crop) (Klapsas et al., 2022), and prosodic shifts (Klapsas et al., 2022). Robustness and speaker-invariance require combining multiple augmentation types, as ablation shows EERs worsening >2x without noise/reverb (Lepage et al., 2022).

3. Positive Sampling, Clustering, and Pseudo-label Generation

Positive pair definition is central. Conventional SSL forms pairs from within the same utterance—effective at removing content but often confounded by channel. The SSPS framework (Lepage et al., 20 May 2025) generalizes positives: for each anchor, a pseudo-positive is sampled from a distinct utterance assigned to the same (or neighboring) cluster via k-means on a frozen memory bank. This reduces channel bias and lowers intra-speaker variance, yielding >58% EER reduction over naive positives.

Iterative bootstrapping (Cai et al., 2020, Singh et al., 2020) alternates: (1) clustering embeddings to generate pseudo-labels, with cluster purification/filtering, and (2) training a new embedding/classification network supervised by these labels. Over multiple rounds, NMI with ground-truth speakers and EER improve steadily until saturation. Prototypical memory banks and "class-collision" correction further resolve same-speaker, cross-recording positives (Zhang et al., 2022, Xia et al., 2020).

Contemporary frameworks for multi-talker ASR and diarization incorporate graph-based clustering (PIC), path integral affinities, or neural PLDA metric learning with joint optimization for both embeddings and metric under self-supervised binary cross-entropy over pseudo-labels (Singh et al., 2021).

4. Loss Functions and Training Objectives

Canonical self-supervised speaker embedding losses are:

• Contrastive (InfoNCE/SimCLR/MoCo): Emphasize positive alignment/intra-speaker invariance, negative repulsion/inter-speaker separation, often with large queues or memory banks:

$$\mathcal{L}_{\text{contrast}} = -\log \frac{\exp(\cos(\mathbf{z}_i, \mathbf{z}_i')/\tau)}{\sum_j \exp(\cos(\mathbf{z}_i, \mathbf{z}_j)/\tau)}$$

(Xia et al., 2020, Lepage et al., 2022).

• Non-contrastive/distillation (BYOL, DINO): No negatives; force alignment of online prediction and stopped-gradient target, prevent collapse via predictor:

$$\mathcal{L}_{\mathrm{BYOL}} = \| \overline{p}_\theta - \mathrm{stopgrad}(\overline{z}'_\xi) \|_2^2$$

(Zhang et al., 2022, Klapsas et al., 2022).

• Prototype/Memory-NCE: Introduce clustering prototypes as additional positives, reducing class-collision (Xia et al., 2020, Zhang et al., 2022).
• Information Maximization/Regularization: Barlow Twins, VICReg, and similar losses directly penalize redundancy and variance collapse, e.g.,

$$\mathcal{L}_{\mathrm{VICReg}} = \frac{1}{N} \sum_{i} \| \mathbf{z}_i - \mathbf{z}_i' \|_2^2 + \mu v(\mathbf{Z}) + \nu c(\mathbf{Z})$$

(Lepage et al., 2022).

• Bootstrap Equilibrium + Uniformity: Predict one view from another (via predictor and EMA target) with a uniformity regularizer that spreads embeddings on the sphere to avoid collapse, independent of negatives (Mun et al., 2021).
• Adversarial Augmentation Invariance: Augmentation adversarial loss penalizes the encoder if the discriminator can detect which augmentation (channel) was applied (Huh et al., 2020).

Combining these losses—e.g., VICReg at one network stage, InfoNCE at another—has been found to outperform single-objective models (Lepage et al., 2022).

5. Practical Implementations and Quantitative Benchmarks

Recent advances have substantially closed the gap between self-supervised and fully supervised systems in both speaker verification (SV) and speaker diarization (SD).

System & Loss	Backbone	EER (%) VoxCeleb1-O	SOTA ref.
Supervised x-vector	ResNet-34	1.51	(Cai et al., 2020)
C3-DINO	ECAPA-TDNN	2.2	(Zhang et al., 2022)
SSPS (SimCLR)	ECAPA-TDNN	2.57	(Lepage et al., 20 May 2025)
DELULU	Transformer	13.53 (zero-shot) / 5.63 (finetune)	(Baali et al., 20 Oct 2025)
WavLM+MHFA w/ SSL PL	WavLM-base+	0.99	(Miara et al., 2024)
Bootstrap Equil. + MLS	Fast ResNet	6.42	(Mun et al., 2021)

Key observations:

• Iterative pseudo-labeling (e.g., (Cai et al., 2020, Miara et al., 2024)) yields monotonic EER improvements (e.g., 8.86→3.45% over 5 rounds).
• Pseudo-positive sampling (SSPS) closes >75% of the gap to oracle positives (1.72% EER) for SimCLR.
• Deep clustering with self-supervised metric learning reduces DER by up to 60% over x-vector-PLDA-AHC diarization (Singh et al., 2021).
• Information maximization frameworks (VICReg, Barlow Twins) outperform pure contrastive or cross-entropy objectives, especially when fine-tuned with minimal labels (Lepage et al., 2022).

Ablation studies confirm the criticality of data augmentation (removal leads to >2× EER), variance/covariance regularization (prevents collapse), and large memory or clustering banks for positive selection.

6. Advanced Topics and Recent Extensions

Recent research trends include:

• Cross-modal self-supervision: Joint training on audio and face video (using cross-modal matching and disentanglement losses) yields robust speaker identity embeddings that outperform supervised training in low-label regimes (Nagrani et al., 2020).
• Speaker-conditioned SSL for multi-talker ASR: Conditioning models such as HuBERT/WavLM on enrollment speaker embeddings via conditional layer-norm (CLN) substantially boosts ASR WER in overlaps or mixtures, illustrating transfer across domains (Huang et al., 2022).
• Uncertainty-aware embeddings: Probabilistic embeddings via learned per-utterance covariance parameters support confidence-calibrated SV with mutual likelihood scores, improving minDCF (Mun et al., 2021).
• External supervision at the pseudo-label stage: DELULU employs frame-level embeddings from a pretrained speaker verification model in k-means cluster assignment, introducing speaker-discriminative bias at the clustering step, outperforming acoustic-only clusterings (Baali et al., 20 Oct 2025).
• Non-contrastive learning and class-collision mitigation: Hybrid approaches employ contrastive learning with class-collision correction followed by negative-free DINO training, with large batch/teacher heads, achieving state-of-the-art for SSL SV (EER 2.2%) (Zhang et al., 2022).
• End-to-end fine-tuning of ASR backbones: WavLM-based models fine-tuned with SSL pseudo-labels via AAM-Softmax nearly match fully supervised SV at scale, especially when pseudo-labels are refined via clustering/fine-tuning cycles (Miara et al., 2024).

7. Current Limitations and Future Directions

Despite remarkable progress, several challenges and opportunities remain:

• Quality of pseudo-labels: All clustering-based methods are sensitive to the initial cluster quality and are susceptible to label noise; purification strategies, confidence filtering, and cluster regularization are active research areas (Cai et al., 2020, Singh et al., 2020).
• Channel variability: Standard SSL frameworks often conflate speaker and channel; SSPS and adversarial training explicitly decouple these signals (Lepage et al., 20 May 2025, Huh et al., 2020).
• Scalability and memory: Large memory banks, queues, or full-dataset clustering present scalability limits for extremely large corpora.
• Fine-tuning with minimal supervision: Approaches that mix SSL with small labeled subsets (semi-supervised) can surpass fully supervised results in low-label regimes (Lepage et al., 2022, Nagrani et al., 2020).
• Generalization across domains: External supervision at the clustering stage (DELULU) or mixture-aware pre-training (WavLM+) increases robustness on profiling and unseen domains (Baali et al., 20 Oct 2025, Huang et al., 2022).
• Unified architectures: There is increasing interest in universal encoders for both content and speaker-aware processing, leveraging multi-task/self-supervision and modality integration.

A plausible implication is that further advances may arise from dynamic pseudo-label refinement, hybrid objectives (combining open-set metric learning, prototype memory, and distillation), and leveraging cross-modal and multi-view data for disentangled, domain-robust speaker embedding learning.