Audio-Only SSL Framework

• Audio-only SSL frameworks are self-supervised pipelines that learn from unlabeled audio data by using techniques such as masked prediction, contrastive learning, and clustering.
• They employ advanced architectures like CNNs and Transformers with input representations (e.g., log-Mel spectrograms) to extract robust embeddings for speech, music, and environmental sound tasks.
• These frameworks integrate diverse loss functions and ensemble strategies to optimize performance, achieving state-of-the-art results on benchmarks like ESC-50 and AudioSet.

An audio-only SSL (self-supervised learning) framework refers to a machine learning pipeline designed to learn meaningful representations from large quantities of unlabelled audio data without relying on any visual, text, or multimodal information. These frameworks employ self-supervised objectives that exploit the structure within the audio waveform or its derived representations (e.g., log-Mel spectrograms) to learn robust embeddings useful for a broad spectrum of downstream audio tasks, including speech, music, and environmental sound classification. The core principle is to devise auxiliary or pretext tasks—such as masked prediction, contrastive learning, or clustering—using only the audio modality, enabling models to utilize vast unlabelled corpora in a label-efficient and domain-general way.

1. Foundational Principles and Objectives

Audio-only SSL frameworks are motivated by the need for universal, label-efficient representations covering the heterogeneity of audio signals, from speech to complex acoustic events. The objectives fall into three main categories:

• Predictive modeling: learning by predicting future, masked, or reordered segments of audio, e.g., auto-regressive predictive coding (APC), masked predictive coding (MPC), and masked spectrogram modeling (such as in TERA or DAPC).
• Contrastive learning: learning by associating paired local or global augmentations of the same audio while discriminating against negatives, e.g., SimCLR-Audio, CPC, wav2vec 2.0, and instance discrimination objectives, typically with InfoNCE loss.
• Clustering/grouping-based approaches: assigning latent cluster (k-means) or vector-quantized (VQ) codes to representations and formulating discrete prediction or consistency objectives, e.g., HuBERT, BEATs, and more recently, Masked Modeling Duo with configurable pretext plus domain-specific branch (Liu et al., 2022, Chen et al., 2022, Niizumi et al., 2024).

The ultimate goal is to create general-purpose feature extractors whose embeddings are transferable to both speech and non-speech tasks, addressing deficiencies in models trained with only one audio domain in mind.

2. Framework Architectures and Feature Pipelines

Contemporary audio-only SSL frameworks adopt architectural innovations from the vision and speech literature to efficiently process sequential audio data. The canonical frameworks are summarized below:

Framework	Backbone Architecture	Input Representation	Key SSL Objective
wav2vec 2.0/Hubert	1D CNN + Transformer	16 kHz raw waveform	Masked contrastive/clustering
BEATs	ViT Transformer + acoustic tokenizer	16x16 Mel patches	Masked discrete token prediction
Dasheng	Masked Transformer Encoder-Decoder	Mel spectrogram chunks	Masked autoencoding (MAE)
M2D/M2D-X	ViT-based encoder (duo)	Log-mel, non-overlap patches	Masked momentum dual encoding + task-specific branch
USAD	Transformer, sparse teacher distill.	128-bin Mel, 25 ms hop	Layer-to-layer matching from SOTA teachers
SPEAR	Zipformer (multi-res) + MVQ	128-bin Mel, conv downsampling	Masked prediction of N-way codebook tokens

Architectures typically involve:

• Either convolutional front-ends (EfficientNet, CNN, ResNet-18) or ViT-style/Transformer encoders for sequence modeling (Chen et al., 2024, Niizumi et al., 2024, Chang et al., 23 Jun 2025, Yang et al., 29 Oct 2025).
• Intermediate patch or chunk embeddings, e.g., 16x16 Mel, 40 ms Mel chunks at 25 Hz, or patch-based representations for ViT-style models.
• For frameworks such as Dasheng or AudioMAE, a lightweight decoder module is used during pre-training only, whereas in teacher-student or token prediction frameworks, only the encoder is kept at inference (Chen et al., 2022, Dinkel et al., 2024, Niizumi et al., 2024, Yang et al., 29 Oct 2025).

3. Self-Supervised Objectives and Loss Functions

The diversity of SSL objectives in audio-only frameworks is aligned with the nature of the representation to be learned:

• Masked prediction losses: Models such as Dasheng, M2D, BEATs, and SPEAR leverage high mask ratios (60–80%) and train encoders to predict either continuous feature targets (e.g., Mel spectrogram patches in Dasheng and M2D) or discrete codebook tokens (in BEATs, SPEAR) at masked positions (Dinkel et al., 2024, Niizumi et al., 2024, Chen et al., 2022, Yang et al., 29 Oct 2025).
• Contrastive objectives: InfoNCE/NT-Xent losses push apart representations from different audio segments while pulling together augmented versions of the same audio, often in conjunction with SpecAugment, noise, or time-stretching (Emami et al., 2021, Liu et al., 2022).
• Hybrid and layer-matching objectives: Recent works such as USAD employ a sparse layer-to-layer distillation loss—L₁ distance and negative log-sigmoid cosine similarity—between student and teacher encoder features at selected layers, avoiding contrastive negatives and promoting task-agnostic transfer (Chang et al., 23 Jun 2025).
• Federated and aggregation-aware SSL: FASSL demonstrates that objective choice (clip prediction, SimCLR, Barlow Twins) interacts non-trivially with federated aggregation and data partitioning, impacting robustness in decentralized training (Rehman et al., 2024).

SSL loss functions are typically constructed to be differentiable w.r.t. the encoder parameters and designed to drive clustering, abstraction, or invariance to nuisance acoustic factors.

4. Ensemble and Fusion-Based Frameworks

A notable approach in the field is the construction of ensemble frameworks which fuse the complementary properties of multiple powerful SSL models trained on audio:

• The NTU-GURA ensemble (Wu et al., 2022) fuses wav2vec 2.0 Large, HuBERT xLarge, and CREPE using feature alignment (interpolation) followed by either feature averaging or, more effectively, concatenation.
• Intra-model aggregation via layer averaging for transformer backbones (e.g., HuBERT, wav2vec 2.0) improves downstream transfer, with ablations confirming layer aggregation’s importance for generalization.
• Concatenating latent features (rather than averaging) preserves the distinct strengths of phonetic, spectral, and pitch-focused models, enabling state-of-the-art results on diverse benchmarks (e.g., ESC-50: 73.4% fusion vs. 60.3% HuBERT; NSynth pitch: 84.6% fusion vs. 40.2% wav2vec 2.0) (Wu et al., 2022).

This ensemble fusion strategy is motivated by the distinct but complementary information captured by each backbone, with performance gains seen across speech, sound, and music tasks.

5. Practical Training Protocols and Evaluation

Audio-only SSL frameworks employ large-scale, domain-diverse unlabelled datasets spanning tens to hundreds of thousands of hours: AudioSet, VGGSound, FSD50K, MTG-Jamendo, and specialized corpora for speech or music (Dinkel et al., 2024, Chang et al., 23 Jun 2025, Yang et al., 29 Oct 2025).

Key protocol elements include:

• High mask ratios (typically >60%) to enforce the learning of robust contextual dependencies.
• Data augmentations: temporal masking, frequency masking, time-stretching, noise injection, room impulse response convolution, and mixup are widely used, depending on the framework (Emami et al., 2021, Niizumi et al., 2024, Liu et al., 2022).
• Pre-training via distributed data-parallel SGD (e.g., AdamW, ScaledAdam), with batch sizes of up to 2048 samples, builds representations suitable for transfer.
• Evaluation proceeds on frozen encoders, with shallow classifiers (typically 1–2 layer MLPs) attached for downstream tasks such as ESC-50, AudioSet-20K, SUPERB, HEAR, with benchmarks reporting mAP, accuracy, and F1 for cross-domain comparison (Chen et al., 2022, Dinkel et al., 2024, Niizumi et al., 2024, Chang et al., 23 Jun 2025, Yang et al., 29 Oct 2025).

Performance ablations are critical: masking ratio, patch/chunk size, loss balancing (global vs frame regression), and layer aggregation are all shown to be major contributors to final encoding quality.

6. Limitations, Extensions, and Future Directions

Despite substantial progress, several limitations and open questions remain:

• Certain frameworks, notably those using only speech-SSL backbones or naive averaging fusion, are weak on fine-grained music tasks or tonal genre classification—targeted music expertise (via CREPE or dedicated codebooks) improves these results but does not solve domain adaptation completely (Wu et al., 2022).
• Most current architectures rely on Mel-spectrogram input rather than learning directly from the raw waveform, though some research explores (1D-)CNN-based or SincNet front-ends for this purpose (Liu et al., 2022).
• Ensemble/fusion methods have yet to explore fully adaptive weighting or attention-based late fusion. This suggests further exploration of learned fusion modules, gating, or cross-attention alignment at feature or intermediate hidden levels could yield further gains (Wu et al., 2022).
• Masking schedules and the granularity of discrete tokenization (BEATs, SPEAR) are active areas; multi-codebook VQ appears beneficial for fine-grained audio event modeling (Yang et al., 29 Oct 2025).
• Federated SSL and privacy-preserving training protocols, as demonstrated by FASSL, operate at parity with centralized regimes for general-purpose retrieval and classification, indicating scalability and deployability in privacy-constrained scenarios (Rehman et al., 2024).
• Recent universal frameworks such as USAD, Dasheng, and SPEAR demonstrate that scaling both model parameters and dataset diversity is essential for optimal transfer to varied downstream tasks (Chang et al., 23 Jun 2025, Dinkel et al., 2024, Yang et al., 29 Oct 2025). A plausible implication is that continual and multi-domain learning, leveraging specialized teacher models (USAD) and multi-branch distillation, will further bridge the generalization gap.

7. Summary of Key Results and Design Insights

Audio-only SSL frameworks have achieved impressive performance, with the best models and ensemble/fusion systems now exceeding or matching supervised and cross-modal baselines on a wide range of tasks:

Model or Framework	AudioSet-20K mAP	ESC-50 Acc	HEAR avg.	Remarks
BEATs₍iter3⁺₎	38.9	98.1	—	SOTA masked token SSL, iter. trainer
EAT (ViT-B)	40.2	95.9	—	Efficient block-masking and reg.
Dasheng-1.2B	—	—	81.3	Large-scale MAE (272 k h, 1.2 B params)
M2D-X	41.8	97.2	~80.6	Universal, adaptive, robust to noise
NTU-GURA ensemble	—	73.4	—	Outperforms all single backbones
SPEAR (Large)	49.8	—	79.18	Audio-only, MVQ token, Zipformer
USAD (Base)	35.7	91.1	78.7	Jointly distilled from speech/audio

The interplay between architecture, loss formulation, masking strategy, and data scale is central to the ultimate utility of audio-only SSL encoders. Research continues to expand the landscape of universal, efficient, and robust self-supervised methods for audio understanding, with frameworks such as Dasheng, M2D/M2D-X, USAD, and SPEAR currently representing the state of the art (Wu et al., 2022, Chen et al., 2022, Niizumi et al., 2024, Dinkel et al., 2024, Chang et al., 23 Jun 2025, Yang et al., 29 Oct 2025, Roy et al., 3 Jul 2025, Chen et al., 2024, Rehman et al., 2024).