Speech Maximum Mean Discrepancy (SMMD)

• SMMD is a quantitative framework that evaluates how self-supervised speech models capture speaker-specific features by probing fixed-layer representations.
• It leverages systematic layer-wise probing using a lightweight MLP to assess discriminability, achieving peak accuracies of 90.2% for pitch, 72.0% for tempo, and 78.5% for energy.
• The analysis informs optimal feature extraction for downstream tasks such as speaker verification, TTS, and demographic prediction, while demonstrating performance gains over comparable models.

Speech Maximum Mean Discrepancy (SMMD) refers to a quantitative approach for probing and evaluating the extent to which self-supervised speech models, such as WavLM Base+, encode speaker-specific features, notably pitch, tempo, and energy, across their internal representations. Although the specific term "SMMD" does not explicitly appear in the referenced literature, the underlying concept is present: feature-specific probing using fixed representations derived from internal layers of large-scale speech SSL models to measure discriminability and feature selectivity. In particular, these methods enable systematic analysis of how speaker-related acoustic and prosodic information is hierarchically captured and separated from linguistic content within models like WavLM Base+ (Chiu et al., 9 Jan 2025, Yang et al., 17 Feb 2025, Chen et al., 2021).

1. Model Architecture and Representation Extraction

The WavLM Base+ architecture consists of a 7-layer 1D convolutional feature encoder that processes raw waveforms sampled at 16 kHz, followed by a stack of 12 Transformer blocks with gated relative position bias. Each Transformer block operates with a hidden size of 768 and 8 attention heads. Input speech is transformed to a series of latent feature vectors, which are then refined by the Transformer to produce hidden states $$H^{(l)} = [h^{(l)}_1, \dots, h^{(l)}_T] \in \mathbb{R}^{T \times 768}$$ at each layer $l$ (Chiu et al., 9 Jan 2025, Chen et al., 2021).

Feature vectors across the time dimension are typically aggregated via average pooling: $$\bar{h}^{(l)} = \frac{1}{T} \sum_{t=1}^T h^{(l)}_t,$$ while the utterance-level embedding used in downstream tasks is

$e = \frac{1}{L} \sum_{t=1}^L h_t,$

where $L$ denotes the number of frames after convolutional downsampling (Yang et al., 17 Feb 2025).

2. Probing Methodology for Speaker-Specific Features

Layer-specific evaluation of encoded features employs the freeze-and-probe paradigm. Specifically, each $\bar{h}^{(l)}$ from WavLM Base+ is passed to a fixed-parameter lightweight probe: a single-hidden-layer multilayer perceptron (MLP) with 500 hidden units. The output is a three-class prediction $\hat{y} \in \mathbb{R}^3$ representing discretized tertiles—low, normal, high—of pitch, tempo, or energy: $$z = W_2 \cdot \mathrm{ReLU}(W_1 \bar{h}^{(l)} + b_1) + b_2, \quad \hat{y} = \mathrm{softmax}(z),$$ with only probe weights $\{W_1, b_1, W_2, b_2\}$ trainable. The cross-entropy loss over the N-sample batch is: $$\mathcal{L} = -\frac{1}{N}\sum_{i=1}^N \sum_{c \in \{\text{low},\text{norm},\text{high}\}} \mathbf{1}\{y_i = c\} \log \hat{y}_{i,c}.$$ Features are measured via standard DSP tools (e.g., Praat), and then discretized globally for target assignment (Chiu et al., 9 Jan 2025).

3. Quantitative Results: Layer-wise and Model Comparisons

Comprehensive evaluation reveals acute hierarchical specialization across WavLM Base+ layers. Probing results indicate:

Feature	Peak Layer ($l^*$)	Accuracy at $l^*$ (%)
Pitch	9	90.2
Tempo	7	72.0
Energy	8	78.5

Pitch discrimination accuracy increases steadily, peaking at layer 9 before declining, indicating late-middle layers encode this property most distinctly. Tempo peaks earlier (layer 7), and energy accuracy peaks near the transition from lower to mid-layer (layer 8). For comparison, WavLM Base+ exceeds HuBERT-Base, Wav2Vec2-Base, and WavLM-Base in peak accuracy for all three features by 1–2 percentage points (Chiu et al., 9 Jan 2025).

Model	Pitch (%)	Tempo (%)	Energy (%)
HuBERT-Base	87.5	70.1	76.3
Wav2Vec2-Base	85.9	69.8	74.9
WavLM-Base	88.3	70.5	77.2
WavLM-Base+	90.2	72.0	78.5

4. Hierarchical Feature Encoding and Funnel Structure

Transformer-based speech SSL models such as WavLM Base+ display hierarchical encoding of speaker-specific attributes. Early layers (1–4) encode low-level spectral and amplitude cues, mid-layers (5–8) specialize in para-linguistics (prosody, rhythm, speaker timbre), and upper layers (10–12) focus on linguistic abstractions. The decline in probe accuracy at the final layers correlates with the model's shift towards representation of content for ASR tasks. This structure can be summarized as a "funnel": broad, undifferentiated acoustic encoding in lower layers; selective para-linguistic discrimination at mid-depth; refined phonetic/linguistic information at the apex (Chiu et al., 9 Jan 2025).

5. Implications for Downstream and Demographic Applications

Analysis of layer-wise feature encoding informs optimal extraction points for downstream applications. For instance, speaker verification benefits from embeddings extracted around layers 8–9, maximizing pitch and energy discrimination without full fine-tuning. Text-to-speech and voice conversion tasks requiring rate manipulation target tempo-discriminative layers (notably layer 7). Combining mid-layer para-linguistic and upper-layer linguistic features enhances expressive synthesis and emotion recognition.

For demographic prediction, WavLM Base+ embeddings—average pooled over the top Transformer layer, yielding 768D vectors per utterance—capture both acoustic-phonetic and prosodic information correlated with attributes such as age, gender, and native language. Performance benchmarks indicate strong results, e.g., mean absolute error of 4.94 for age prediction and >99.8% accuracy for gender classification (Yang et al., 17 Feb 2025). The approach facilitates plug-and-play integration: pretrained, frozen WavLM models with lightweight supervised heads outperform i-vector and x-vector baselines on demographic tasks with high efficiency.

6. Pre-training Objectives, Data, and Robustness Enhancements

WavLM Base+ pre-training combines masked speech prediction—where input spans are replaced by a trainable mask embedding and targets are derived from k-means clustering on feature representations—with denoising and multi-speaker simulation. For 20% of training, input is replaced by noise-corrupted or mixed-speech signals at controlled SNR, promoting robustness to interference and environmental variation. Ablation studies confirm each augmentation (denoising, gated relative position bias) is critical to achieving state-of-the-art benchmark performance on tasks involving speaker identity and separation (Chen et al., 2021).

Model scaling from the original 960 h LibriSpeech to the 94,000 h “Mix 94k h” corpus (Libri-Light, GigaSpeech, VoxPopuli) further contributes to generalization and transfer to speaker-sensitive applications (Chen et al., 2021).

7. Evaluation Metrics and Best Practices

Feature probing employs macro-averaged accuracy and F1-scores for quantifying discrimination across low/normal/high bins. Downstream demographic tasks use mean absolute error (age) and classification accuracy (gender, language, education). Best practices include freezing WavLM weights to avoid overfitting, employing average pooling for compact utterance embeddings, and training only lightweight supervised heads. For efficiency, embeddings can be pre-computed, and learning-rate scheduling on the head stabilizes training across heterogeneous datasets (Yang et al., 17 Feb 2025).

---

In summary, the probing framework for speaker-specific features—operationalized via frozen self-supervised WavLM Base+ representations and lightweight supervised probes—quantifies hierarchical encoding of pitch, tempo, and energy, thus enabling fine-grained control and targeted extraction for speaker-centric tasks, demographic attribute prediction, and para-linguistic analysis in speech processing (Chiu et al., 9 Jan 2025, Yang et al., 17 Feb 2025, Chen et al., 2021).