Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models

Abstract: Audio-LLMs (ALMs) have shown remarkable success in zero-shot audio classification by aligning audio waveforms with text. Recent efforts to improve downstream performance focus on learning optimal text prompts. However, previous approaches focus on the text encoder, leaving the potential of learnable prompts within the audio encoder unexplored. In this paper, we propose a novel framework that introduces trainable prompts into the audio encoder to capture task-specific acoustic features. We demonstrate that integrating audio-side prompt learning with existing text-side approaches enhances few-shot adaptation. Through extensive experiments across 11 datasets show that integrating our method as a plug-and-play module alongside existing text prompt tuning generally leads to performance improvements. These findings suggest that explicitly modulating the audio representation space effectively complements text-only prompting approaches. The code is available at https://github.com/hyebin-c/aspl.

• The paper presents ASPL, a novel framework that introduces stage-wise acoustic modulation to overcome text-centric biases in audio language models.
• It employs prompt-driven affine modulations at spectral, token, and transformer stages, achieving significant accuracy gains with minimal parameter cost.
• Experimental results across 11 classification tasks show that ASPL improves few-shot performance with average top-1 accuracy gains of over 1%.

Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio LLMs

Introduction and Motivation

The paper "Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio LLMs" (2606.15751) addresses a key limitation in current prompt-based adaptation methods for audio-LLMs (ALMs): the text-centric bias of prompt learning. While numerous works optimize deep text encoders with learnable prompts to enhance ALM adaptation for downstream few-shot tasks, they largely neglect the modulation of the audio encoder. This approach restricts cross-modal adaptation and limits performance, especially when there is a domain mismatch between the large-scale pretraining corpus and target audio distributions. The authors propose Audio-Side Prompt Learning (ASPL), a framework injecting lightweight, learnable, stage-wise modulations at critical points in the audio pipeline, forming a plug-and-play component that complements traditional text-side prompt learning.

Methodology: Audio-Side Prompt Learning (ASPL)

ASPL is built atop the CLAP-HTSAT architecture and introduces prompt-driven affine modulations at three strategic locations within the audio encoder: (1) spectral (log-mel spectrogram) transformation, (2) patch embedding output, and (3) early Swin Transformer blocks. These locations are chosen to systematically recalibrate both low-level and intermediate representations, remedying intra-class variability and adaptation challenges endemic to audio signals.

Figure 1: Overview of the ASPL framework with three-stage audio encoder modulation and integration with text-side methods.

Unlike text-side prompting such as CoOp, CoCoOp, or PALM, which prepend or transform textual context tokens, ASPL deploys continuous, shared affine parameters ($\gamma$, $\beta$) applied channel- or frequency-wise. This design provides strict parameter efficiency: the number of trainable parameters is invariant to label space size, scaling only with embedding dimensions at the specific encoder stages. ASPL thus addresses both the adaptability and efficiency bottlenecks of ALM prompt tuning.

Experimental Evaluation

Datasets and Protocol

The evaluation covers 11 classification tasks, spanning instrument, event, emotion, and scene classification benchmarks. Following the PALM protocol, few-shot scenarios with $K$ in $\{1, 2, 4, 8, 16\}$ samples per class are considered, with average top-1 accuracy evaluated across multiple random seeds.

Main Results

ASPL yields consistent performance benefits when incorporated as a plug-in to state-of-the-art text-prompting methods, with an average improvement in accuracy of approximately 1.1% under 16-shot settings, and absolute gains exceeding 1.4% in integrated PALM-based models. Importantly, these improvements are achieved with only marginal increases in parameter count (e.g., 320 for ASPL and 704 for ASPL$^*$) and negligible inference latency, supporting the method’s suitability for real-time or resource-constrained deployment.

The results further highlight a synergy between ASPL module depth and text-side prompt dynamism:

• Dynamic prompt pairing (e.g., CoCoOp): Shallow ASPL is optimal due to built-in instance-level variance handling in the text branch.
• Static/class-specific prompts (e.g., CoOp, PALM): Deep ASPL$^*$ is necessary to compensate for the rigidity of text encoders, yielding maximal gains via early Swin block conditioning.

Ablation Studies

Impact of Shot Availability

ASPL maintains a clear lead over all text-only baselines across shot counts $K \geq 2$, with only minor degradation under 1-shot conditions, attributable to limited supervision for the acoustic prompt parameters. The hierarchical, multi-stage modulation strategy outperforms naive single-stage prompting, confirming that coordinated recalibration yields stable performance across increasing supervision regimes.

Figure 2: Few-shot ablation—ASPL provides robust adaptation across all backbone methods and shot counts, outperforming text-only counterparts.

Location and Parameterization Effects

Isolated modulations at either spectral or token level are suboptimal, often degrading performance due to disruption of representational flow. In contrast, joint modulation at spectral, token, and early structural (Transformer block) stages achieves the highest accuracy. Attempted adaptations of text-side output space prompting to audio encoders are ineffective, reinforcing the uniqueness of ASPL’s architectural rationale for audio data.

Implications and Future Directions

Theoretical: ASPL establishes the utility of dual-modality prompt learning for ALMs, directly addressing the inherent instability and continuous nature of audio signals. The results challenge the prevailing orthodoxy that only text encoders are suitable for efficient prompt-based adaptation and support the necessity of coordinated encoder modulation for effective cross-modal transfer.

Practical: Extreme parameter efficiency, hardware-agnostic deployment feasibility, and versatility as a plug-in for a range of text-side methods make ASPL highly relevant for edge inference, domain-robust ALM deployment, and few-shot/continual learning pipelines.

Looking ahead, the framework paves the way for future dynamic gating mechanisms that can automatically adjust modulation depth for optimality across tasks and domains. Integrating sample-dependent or context-aware modulation (e.g., via meta-learning controllers) could further enhance ALM adaptability and cross-domain robustness.

Conclusion

ASPL introduces stage-wise, lightweight, and highly efficient acoustic prompting, overcoming the intrinsic limitations of text-centric prompt learning in ALMs. Through joint, early-stage encoder recalibration, ASPL consistently and robustly improves few-shot classification performance while imposing negligible computational cost, marking a significant advancement in practical, scalable ALM adaptation strategies. This work establishes a foundation for future modular, dual-sided adaptation of foundation models across multimodal domains.