Image-based Speaker Embedding

• Image-based Speaker Embedding (ISE) is a method that uses visual information from faces or lips to create robust speaker representations.
• It employs cross-modal transfer techniques, such as target, relative, and clustering transfer, to align audio and visual data with enhanced triplet loss strategies.
• ISE methods achieve quantifiable improvements in tasks like speaker verification and diarization, even under noisy or incomplete audio-visual conditions.

Image-based Speaker Embedding (ISE) refers to computational representations of a speaker’s identity, where visual information—most notably face or lip images—serves as a supervision signal, auxiliary feature, or even a direct source of the embedding. ISE approaches exploit cross-modal relationships between facial and vocal information, leveraging inherent correlations in identity, and can regularize or enhance audio-domain speaker embeddings. These embeddings are increasingly central to tasks requiring robust speaker modeling in settings with limited audio data, significant noise, or the need for cross-modal person analysis, including speaker diarization, cross-modal retrieval, and target speaker extraction (Le et al., 2017, Pan et al., 2022).

1. Cross-modal Transfer Learning for Audio Embeddings

A principal line of work applies transfer learning by exploiting embeddings learned from large-scale face datasets, under the assumption that voice and face share entangled latent attributes such as age, gender, and ethnicity. Given (i) a pretrained face-embedding network $f^V$ mapping face images $x^V$ to a $d$-dimensional unit hypersphere, and (ii) a speaker-turn embedding network $f^A$ to be learned for short audio segments $x^A$, the objective is to regularize $f^A$ using structural information encoded in the face manifold.

Three transfer strategies are established (Le et al., 2017):

• Target Embedding Transfer: The face embedding for each identity serves as a "teacher" target. When paired face and voice samples exist, the approach minimizes cross-modal embedding distance for the same identity and maximizes disparity for distinct identities, operationalized via multimodal triplet loss.
• Relative Distance Transfer: Instead of aligning audio and visual embeddings exactly, the method enforces the relative proximity structure in the face embedding to be preserved in the audio domain. For example, if the mean face vector of identity $j$ is closer to $i$ than to $k$, the same should be true for their audio embeddings.
• Clustering Structure Transfer: K-means clustering is applied to mean face representations to discover latent groups (e.g., gender or age clusters). Cluster assignments provide "soft" labels, guiding the audio model to group speaker embeddings based on visual-cluster membership, encouraging similarity within and disparity across clusters.

Regularization strategies above are formalized using Euclidean distances on the unit hypersphere in combination with margin-based triplet losses.

2. Embedding Space Architectures and Training

Face and speech embedding networks employ domain-specific architectures:

• Face Embedding ($f^V$):
  • Base: ResNet-34. Pretrained for multi-class face identification on CASIA-WebFaces (~0.5M images, 10,575 classes).
  • Fine-tuned with triplet loss on REPERE dataset (208 train identities).
  • 128-dimensional unit-normalized vectors in $\mathbb{R}^{128}$.
• Speaker-turn Embedding ($f^A$):
  • “TristouNet”: Bidirectional LSTM (hidden size 32), temporal pooling, two fully-connected layers (64→128), final unit-ℓ2 normalization.
  • Inputs: 13 MFCC, energy, $\Delta$, $\Delta\Delta$ features, 10 ms frame shift.

Training employs the RMSProp optimizer ($\text{lr}=1\mathrm{e}{-3}$), mini-batch sampling of hard/semi-hard negatives, and hyperparameter grid search for transfer term weighting ($\lambda \in [0.01, 1]$). Each transfer method is activated separately with a triplet-based loss encoding the specific regularization, though the framework allows their combination.

3. Visual Embedding Extraction for Target Speaker Extraction

In target speaker extraction under audio-visual conditions, image-based speaker embeddings are typically extracted from sequential face/lip frames using a fixed visual front-end:

• Visual Front-Ends:
  • VSR-based (lip-reading): ResNet-18 plus two-layer TCN, producing $D=512$ dimensional embeddings.
  • Speech-lip synchronization network: CNN plus TCN, also $D=512$.

Given a sequence of lip crops $v = (v_1, …, v_T)$ at 25 FPS, the visual network produces $V = [V_1, ..., V_T] \in \mathbb{R}^{D \times T}$, robustly encoding speaker characteristics throughout the utterance (Pan et al., 2022).

4. Embedding Inpainting and Audio-Visual Fusion

Real-world videos are often affected by temporal occlusion or loss of visual signal. Embedding inpainting modules are introduced to reconstruct missing image-based embeddings.

In ImagineNET, multiple speech-mask estimation stages alternate with visual-refinement (inpainting) stages. At each stage, channel-wise concatenation of the speech encoder embedding $X$ and current visual embedding $V^{r-1}$ forms the input to the next mask estimator. Missing visual frames are inpainted by a visual refiner, which is updated in an interlacing procedure between the audio and visual streams.

Two loss functions govern inpainting:

• MSE Reconstruction: $$L_{\mathrm{MSE}}(V, \hat{V}^r) = \frac{1}{T} \sum_{t=1}^T \|V_t - \hat{V}_t^r\|^2$$
• Contrastive InfoNCE: Utilizes cosine similarities/contrasts between recovered and ground-truth embeddings at temperature $\kappa$.

The total multi-stage loss is $$L_\text{total} = L_{\mathrm{SI–SDR}}(s, \hat{y}) + \gamma \cdot \sum_{r=1}^{R-1} L_v(V, \hat{V}^r)$$, where $\gamma$ weights the inpainting loss.

5. Benchmarking and Quantitative Performance

Empirical benefits of ISE methods are evident across multiple audio-visual speech and speaker tasks:

• Speaker Turn Clustering (REPERE, 629 tracks):
  • Speaker-only baseline (TristouNet): Weighted Cluster Purity (WCP) ≈ 0.65, Operator Clicks Index (OCI-$k_\mathrm{min}$) ≈ 275.
  • Target transfer: WCP improved by 3–5 percentage points, OCI-$k_\mathrm{min}$ = 241 (≈12% fewer required clicks).
  • Relative/structure transfer: Moderate gains (OCI-$k_\mathrm{min}$ ≈ 255–256).
• Speaker Verification (ETAPE, 1s segments):
  • TristouNet: Equal Error Rate (EER) = 19.1%.
  • Target/relative/structure transfer: EER in 18.0–18.3% range.
• Target Speaker Extraction with Intermittent Visual Cues (VoxCeleb2-2mix, 51% frames blanked):
  • ImagineNET + MSE: SI–SDR = 10.81 dB, PESQ = 2.894, STOI = 0.854, outperforming TDSE baseline and robust to partial visual absence.
  • ImagineNET benefits remain consistent provided ≥10–20% visual information is present (Pan et al., 2022).

Method	WCP (REPERE)	EER (ETAPE)	SI–SDR (VoxCeleb2-2mix)
TristouNet (audio only)	≈0.65	19.1%	—
Target-transfer	↑3–5%	18.0%	—
Relative/Cluster transfer	~+2%	18.2–18.3%	—
ImagineNET + MSE (51% missing)	—	—	10.81 dB
TDSE (no inpainting, 51% missing)	—	—	9.95 dB

ISE approaches thus yield clear, quantifiable improvements in both clustering and speaker verification under audio-visual settings with limited or degraded data.

6. Analysis of Learned Structure and Cross-Modal Relationships

Alignment to face manifolds provides strong priors about speaker identity, especially when short utterances or scarce data limit classical voice-only methods. Face-based transfers can:

• Anchor speaker embeddings to the well-shaped, discriminative manifold of faces.
• Preserve cluster/ordering relationships, increasing cohesion among identities with similar visual features.
• Reveal groupings that map onto latent attributes (e.g., clusters found by K-means on $\vec{f}^V$ often align with age/gender), with corresponding voice clusters mirroring these properties.

ISE models support bidirectional cross-modal retrieval, with cross-domain queries (face-to-voice and voice-to-face) achieving Precision@K well above chance, indicating formation of a harmonic joint embedding space. A plausible implication is that multimodal regularization endows the speech embedding space with semantic and biometric structure superior to purely audio-optimized projection spaces (Le et al., 2017).

7. Applications and Future Prospects

Image-based speaker embeddings are central in several emerging application domains:

• Speaker diarization in broadcast/video media, especially for processing short speaker turns or overlapping segments where traditional tools like BIC or Gaussian divergence fail.
• Cross-modal indexing, facilitating the "who speaks when" task by exploiting multimedia content at the embedding level.
• Voice–face retrieval and multimedia search, enabling queries such as finding the speaker corresponding to a given face or vice versa.
• Robust target speaker extraction, especially in audiovisual scenarios with intermittent visual cues and noisy mixtures (Pan et al., 2022).

Promising directions include joint/hierarchical incorporation of all three transfer losses, expansion to other domains and languages by leveraging pre-established face manifolds, and the extension of similar regularization frameworks to other biometric signals or more complex/overlapping latent structures.

ISE methods thus represent a convergence of advances in face recognition, speaker representation learning, and multimodal deep learning, offering enhanced robustness and semantic interpretability for high-level speaker analysis in multimedia contexts (Le et al., 2017, Pan et al., 2022).