SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation

Abstract: Human infants, with only a few hundred hours of speech exposure, acquire basic units of new languages, highlighting a striking efficiency gap compared to the data-hungry self-supervised speech models. To address this gap, this paper introduces SpidR-Adapt for rapid adaptation to new languages using minimal unlabeled data. We cast such low-resource speech representation learning as a meta-learning problem and construct a multi-task adaptive pre-training (MAdaPT) protocol which formulates the adaptation process as a bi-level optimization framework. To enable scalable meta-training under this framework, we propose a novel heuristic solution, first-order bi-level optimization (FOBLO), avoiding heavy computation costs. Finally, we stabilize meta-training by using a robust initialization through interleaved supervision which alternates self-supervised and supervised objectives. Empirically, SpidR-Adapt achieves rapid gains in phonemic discriminability (ABX) and spoken language modeling (sWUGGY, sBLIMP, tSC), improving over in-domain LLMs after training on less than 1h of target-language audio, over $100\times$ more data-efficient than standard training. These findings highlight a practical, architecture-agnostic path toward biologically inspired, data-efficient representations. We open-source the training code and model checkpoints at https://github.com/facebookresearch/spidr-adapt.

• The paper introduces a meta-learning framework combining multi-task adaptive pre-training and first-order bi-level optimization to achieve 100× data-efficiency in few-shot speech adaptation.
• It employs episodic active forgetting and interleaved supervision to robustly extract transferable phonemic representations across diverse languages.
• Experimental results show that SpidR-Adapt matches oracle-scale models using only 1 hour of unlabeled adaptation data, outperforming standard SSL baselines.

SpidR-Adapt: Universal Speech Representations for Data-Efficient Few-Shot Adaptation

Motivation and Problem Statement

The efficiency disparity between human infant language acquisition and machine speech models is pronounced: infants internalize basic phonemic structure from only 100–500 hours of natural exposure, whereas current self-supervised learning (SSL) systems, such as HuBERT and WavLM, require over an order of magnitude more data. These models, trained from random initialization and limited by weak inductive priors, fail to robustly capture phonemic contrasts—especially in out-of-domain (OoD) target languages with scarce resources. Addressing this, "SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation" (2512.21204) introduces a biologically inspired meta-learning framework that achieves rapid, data-efficient adaptation to new languages with minimal unlabeled audio.

SpidR-Adapt Framework and Core Contributions

The SpidR-Adapt system instantiates a meta-learning approach built on three primary contributions:

1. Multi-task Adaptive Pre-Training (MAdaPT): Casts few-shot speech representation learning as a bi-level optimization, where the inner loop performs unsupervised adaptation emulating “lifetimes” of new language exposure, and the outer loop meta-optimizes the model for swift cross-lingual adaptation. Episodic sampling and active forgetting—resetting model heads and codebooks at each episode—regularize the induction of transferable speech units.
2. First-Order Bi-Level Optimization (FOBLO): A computationally tractable first-order meta-learning heuristic, replacing second-order derivatives with parameter difference updates between the end of inner-loop SSL adaptation and subsequent supervised steps on labeled data. This preserves efficiency and enables scaling to multi-lingual setups.
3. Interleaved Supervision for Initialization: Alternates self-supervised and phoneme-supervised objectives during initial multi-task pre-training. This initialization, stronger than purely self-supervised alternatives, stabilizes meta-training and yields representations robust to speaker and contextual variation.

The resulting SpidR-Adapt model extends SpidR [Poli2025SpidR], a student-teacher SSL architecture utilizing self-distillation and online clustering, with these meta-adaptive components.

Figure 1: SpidR-Adapt training pipeline showing meta-initialization (interleaved SSL/SL), meta-training (MAdaPT-FOBLO with episodic active forgetting), and unsupervised meta-testing on unseen target languages.

Experimental Protocol and Results

Datasets and Evaluation

Empirical assessment covers adaptation to three test languages (English, French, German) disjoint from the 19 training languages, with adaptation data budgets ranging from 10 minutes to 100 hours. Labeled phoneme data (max 50 hours per source language) is used only for outer-loop meta-optimization and interleaved initialization. Main metrics include ABX (phonemic discriminability; lower is better), spoken LLM (SLM) accuracy (sWUGGY, sBLIMP, tSC; higher is better), PNMI, and PER, all following Zero Resource and Phoneme Discovery Benchmarks.

Data-Efficiency and Phonemic Discriminability

SpidR-Adapt achieves parity with in-domain, oracle-scale models trained on 6,000 hours of target-language speech, using only 1 hour of unlabeled adaptation data—a documented 100$\times$ data-efficiency improvement. The approach outperforms both standard multi-task SSL pre-training and alternative meta-learning heuristics (e.g., Reptile), especially in low-resource adaptation regimes. Interleaved-supervised meta-initialization consistently yields better results than pure SSL initialization, but the major gains stem from MAdaPT-FOBLO meta-training.

Figure 2: On three unseen target languages, SpidR-Adapt (MAdaPT-FOBLO) matches in-domain performance with less than 1 hour adaptation, clearly outperforming both baseline and Reptile meta-learning.

Downstream SLM and Robustness

On downstream SLM metrics—lexical (sWUGGY), syntactic (sBLIMP), and discourse continuity (tSC)—SpidR-Adapt maintains or exceeds oracle baselines, despite minimal adaptation data. Phoneme discovery benchmarks across multiple languages further validate increased mutual information and reduced error rates, with SpidR-Adapt surpassing existing SSL models (e.g., HuBERT) and meta-learning baselines.

Methodological Overview

Bi-level Meta-learning Formulation

Adaptation to a target language $\ell$ is formalized as:

$$\bm{\theta}^*_\ell = \arg\min_{\bm\theta} \mathcal{L}_{ssl}(\bm\theta; \mathbf{D}_\ell^u)$$

with bi-level meta-optimization over source languages set $\mathcal{S}$:

$$\min_{\bm\phi} \; \mathbb{E}_{\ell\sim\mathcal{S}}\left[\mathcal{L}_{sl}\left(\bm\theta^*_\ell(\bm\phi); \mathbf{D}_\ell^s\right)\right] \ \text{where} \;\; \bm\theta^*_\ell(\bm\phi) = \arg\min_{\bm\theta} \mathcal{L}_{ssl}(\bm\theta,\bm\phi; \mathbf{D}_\ell^u)$$

FOBLO provides the update:

$$\bm\phi \leftarrow \bm\phi - \beta \mathbb{E}_{\ell}\left[\bm\theta_\ell^M - \bm\theta_\ell^{M+N}\right]$$

where $M$ and $N$ are the numbers of inner-loop (SSL) and outer-loop (SL) steps, respectively.

Active Forgetting

At the start of each adaptation, SpidR’s prediction heads and codebooks are re-initialized, preventing catastrophic interference and promoting generalization to phonemic contrasts in unseen languages.

Interleaved Supervision

The interleaved SSL/SL initialization alternates between updating on unlabeled multi-lingual data and explicit phoneme classification, supplying inductive bias akin to the early categorical learning seen in infants.

Analysis and Implications

SpidR-Adapt decisively narrows the gap between human-like data efficiency and machine speech learning with compositionally structured, transferable representations, even on entirely unseen languages. The introduction of bi-level episodic meta-learning, especially when coupled with robust initialization and targeted forgetting, demonstrates clear improvements in data efficiency and rapid adaptation. This architecture-agnostic framework suggests a new paradigm for scaling self-supervised learning to low-resource, multilingual setups, holding promise for inclusive spoken AI, transfer learning in language documentation, and unsupervised model adaptation in zero-resource conditions.

On the theoretical front, MAdaPT-FOBLO—with its task-structured episodic design and inductive prior formulation—illustrates the viability of meta-learning for unsupervised representation induction, opening avenues for generalization beyond speech, potentially to vision, multimodal, or continual learning contexts.

Limitations and Future Directions

Despite these advances, SpidR-Adapt still depends on some source language supervision for optimal meta-optimization, and the spoken LLM adaptation is not yet integrated into meta-training. Open challenges remain in scaling further to purely unsupervised setups, integrating discriminative spoken language modeling objectives directly into meta-loops, and extending active forgetting strategies.

Conclusion

The SpidR-Adapt framework demonstrates how meta-learning, informed by biological constraints and advanced with efficient heuristics, can yield universal, data-efficient speech representations. Its substantial improvements in few-shot adaptation suggest a realistic path toward truly low-resource, language-agnostic speech technologies.