CALM: Codified Audio Language Modeling

• CALM is a paradigm that transforms continuous audio into discrete tokens via neural codecs, enabling autoregressive sequence modeling.
• It leverages techniques like residual vector quantization and hierarchical tokenization to achieve high-fidelity and semantically-rich audio representations.
• CALM supports diverse applications, from audio synthesis and classification to multimodal reasoning and few-shot adaptation.

Codified Audio Language Modeling (CALM) is a foundational paradigm in machine hearing that frames audio understanding, generation, and reasoning as autoregressive modeling over sequences of discrete tokens derived from raw waveforms. By codifying continuous audio into compact, semantically-rich token streams—often further aligned to textual abstractions—CALM enables transformer-based models to operate on audio with the same versatility as LLMs do for text. This approach decouples representation learning (via neural codecs or alignment techniques) from sequence modeling, thereby scaling to large, diverse audio domains and supporting multimodal, few-shot, and zero-shot inferences.

1. Formal Definition and Historical Emergence

CALM takes a continuous audio signal $x \in \mathbb{R}^T$ and transforms it into one or more sequences of discrete tokens $z \in \{1,\dots,K\}^{T'}$, with $T' \ll T$, via a neural tokenizer (codec or contrastive module). A decoder, often also neural, can reconstruct a waveform approximation $\hat{x} \approx x$ from the token sequence. The modeling objective is to maximize the joint probability or likelihood over the autoregressive trajectory in token space,

$P(z) = \prod_{t=1}^{T'} P(z_t | z_{<t}).$

This fundamental principle, first operationalized at scale in works such as Jukebox for music and AudioLM for general audio (Castellon et al., 2021, Borsos et al., 2022), abstracts audio modeling to a language modeling task, leveraging the architectural and computational advances established in NLP.

CALM's rapid uptake stems from the fusion of (i) high-fidelity neural codecs for lossy but information-preserving quantization, and (ii) large transformer-based sequence models, including autoregressive and hierarchical approaches. Recent unifications, such as UALM and UniAudio 2.0, further integrate text and audio token streams for tightly-coupled multimodal reasoning (Tian et al., 13 Oct 2025, Yang et al., 4 Feb 2026).

2. Tokenization, Discretization, and Hybrid Representations

The tokenization stage is critical in CALM and manifests in two principal forms:

• Neural Codec Quantization: Systems such as SoundStream, EnCodec, and X-codec employ residual vector quantization (RVQ) to map audio frames into sequences of token indices. Typical implementations split audio into frames (e.g., 20 ms at 16 kHz), with each frame assigned multiple discrete codes by stacked quantizers. The VQ bottleneck is trained using reconstruction loss (e.g., L1), perceptual loss (e.g., multiresolution STFT), and commitment penalties.
• Semantic or Reasoning Token Streams: Recent CALM systems introduce hierarchical or parallel tokenizations. For example, AudioLM combines semantic tokens (via w2v-BERT + k-means) that capture linguistic structure, with acoustic tokens from neural codecs to achieve both long-term coherence and waveform fidelity (Borsos et al., 2022). UniAudio 2.0 factorizes into "reasoning" tokens (aligned to text, low rate) and "reconstruction" tokens (fine acoustic, higher rate) (Yang et al., 4 Feb 2026). The ALMTokenizer employs learnable query tokens and semantic-prior-informed VQ for semantically compact coding (Yang et al., 14 Apr 2025).

A summary table:

Tokenization Method	Semantic Content	Modeling Fidelity
RVQ Codec (SoundStream, EnCodec)	Acoustic structure, timbre	High at higher bitrates
MLM-derived + k-means (AudioLM)	Phonetics, syntax, semantics	Lower, non-invertible
Reasoning + Recon (UniAudio2.0)	Text-aligned + acoustic	Balances both
Query-based (ALMTokenizer)	Holistic, context-aware	Efficient, low bitrate

3. Sequence Modeling Architectures and Training Methodologies

The codified token streams serve as input to powerful sequence models, almost universally based on transformers. Modeling strategies include:

• Autoregressive Language Modeling: Decoder-only transformers, often with hierarchical factorization (AudioLM), directly model the joint distribution over tokenized audio (Borsos et al., 2022, Kreuk et al., 2022).
• Hierarchical and Multi-rate Models: Multi-stage pipelines predict semantic/structure tokens at coarse rates and then condition acoustic token prediction at finer rates, restricting context length per stage and improving training tractability.
• Unified Modalities: UALM and UniAudio 2.0 jointly model text, reasoning, and reconstruction audio tokens in a single autoregressive sequence, sharing all attention and transformer layers, sometimes with modality-specific adapters or layer specialization (Tian et al., 13 Oct 2025, Yang et al., 4 Feb 2026).
• Adapter-based Alignment: For few-shot adaptation, lightweight cross-attention or linear adapters attend to a support set of labeled audio-label pairs, as in Treff/CALM modules (Liang et al., 2023).

Optimization objectives include next-token cross-entropy (standard LM loss), masked prediction tasks (MAE, MLM, MAM), and auxiliary contrastive or semantic-alignment losses where text/audio codes are explicitly aligned (Sachidananda et al., 2022, Yang et al., 14 Apr 2025).

4. Downstream Tasks, Evaluation, and Empirical Results

CALM unlocks a wide spectrum of audio understanding and generation tasks:

• Audio Generation: CALM models, when conditioned on initial prompts or text, can synthesize or continue high-fidelity waveform, as demonstrated by AudioLM, UALM, and Jukebox (Castellon et al., 2021, Borsos et al., 2022, Tian et al., 13 Oct 2025). Hybrid tokenizations enable both structural coherence and timbral fidelity in long-range generation.
• Audio Understanding: Frozen or fine-tuned CALM features yield state-of-the-art results in classification (ESC-50, FSDKaggle18, GTZAN), tagging, emotion recognition, and speaker ID. For example, UniAudio 2.0’s factorized tokens close the gap with continuous SSL features in ASR and event detection (Yang et al., 4 Feb 2026). Treff-based adapters leveraging CALM dynamics significantly improve low-shot adaptation over zero-shot baselines (Liang et al., 2023).

Empirical highlights:

Task/Metric	CALM System	Result(s)
5w5s ESC-50 acc.	Treff (fine-tuned)	75.8% (vs. 54.6% zero-shot, +21.2% absolute) (Liang et al., 2023)
AudioSet MOS	CALM (VQ-VAE + LM)	3.03 (cMOS: 3.18), outperforming CPC/wav2vec baselines (Kreuk et al., 2022)
MIR (average gain)	Jukebox (CALM)	30% improvement over tag-pretrained models (Castellon et al., 2021)
Semantic token gen	AudioLM	ABX error 6.7–7.6%, sWUGGY 71.5% zero-shot (Borsos et al., 2022)
MUSHRA (1.5 kbps)	ALMTokenizer	68.3 (vs. Encodec: 60.1), +8% ESC-50 acc. at equal bitrate (Yang et al., 14 Apr 2025)
Multi-modal chain	UALM	Unified reasoning (audio+text), MOS 3.77–4.01 subjective (Tian et al., 13 Oct 2025)

5. Multimodal Alignment and Contrastive Audio-Language Modeling

CALM frameworks extend beyond pure audio by aligning discrete audio tokens with textual representations:

• Contrastive Pretraining: CALM (Contrastive Aligned Audio-Language Multirate) aligns SpecTran-derived audio tokens with BERT text embeddings, using an NT-Xent dual contrastive loss. This alignment enhances cross-modal consistency and supports flexible downstream fusion (Sachidananda et al., 2022). The multirate nature—drawing positives at both high- and low-temporal rates—enables the model to jointly capture acoustic detail and semantic content.
• Text-Audio Modeling: Recent architectures such as AudioPaLM, UniAudio 2.0, UALM, and LauraGPT implement joint vocabularies and shared transformer backbones for text and audio, supporting tasks ranging from captioning to grounded multimodal dialogue (Tian et al., 13 Oct 2025, Yang et al., 4 Feb 2026). Factorized token streams (reasoning/reconstruction) allow hierarchical modeling and targeted abstraction.

Limitations exist in requiring text data for alignment, sensitivity to overlapping or poorly segmented data, and challenges in extending the paradigm fully unsupervised or to additional modalities (e.g., vision). Proposals for further integration include vision-token fusion and meta-learned temperature/sharpening functions in contrastive or retrieval losses.

6. Challenges, Trade-offs, and Future Directions

Key open challenges in CALM research include:

• Bitrate versus Model Complexity: Lower bitrates reduce sequence length (e.g., with group-residual VQ or semantic query-based coding) and facilitate longer context modeling but may limit high-fidelity reconstruction or task transferability (Yang et al., 14 Apr 2025, Wu et al., 2024). Increasing vocabulary size or codebook depth marginally improves fidelity at the expense of modeling difficulty.
• Semantic-Acoustic Disentanglement: Factorizing token streams (e.g., reasoning vs. reconstruction tokens) supports both understanding (mirroring text abstractions) and high-fidelity generation. Unified interfaces remain an active area of exploration (Yang et al., 4 Feb 2026).
• Scalability and Efficiency: Transformers' quadratic complexity constrains context window length. Approaches including hierarchical modeling, multi-scale attention, or sparsity—implemented in AudioLM, UniAudio—seek to scale to thousands of tokens (Wu et al., 2024, Borsos et al., 2022).
• Evaluation Protocols: Objective metrics (perplexity, SNR, PESQ, MUSHRA, FAD, cMOS) are systematized, but generative audio evaluation remains reliant on human judgment. Transfer, zero-/few-shot, and continual learning benchmarks require further development.
• Multimodal Few-Shot and Reasoning: Adapters like Treff/CALM modules enable few-shot learning in frozen backbones by leveraging small labeled support sets; extension beyond classification to reasoning and sequence generation is ongoing (Liang et al., 2023, Tian et al., 13 Oct 2025).
• Unified Foundation Models: UALM and UniAudio 2.0 demonstrate scaling toward foundation models for audio, capable of generalized multimodal reasoning, generation, and few-/zero-shot adaptation using codified, factorized token streams (Tian et al., 13 Oct 2025, Yang et al., 4 Feb 2026).

7. Significance, Impacts, and Applications

The CALM framework enables a universal paradigm for audio that parallels the progress of LLMs in text. By leveraging codification, unified modeling backbones, and alignment with semantic abstractions, CALM unlocks the following key impacts:

• Universal Audio Intelligence: From robust few-shot classification to text-conditioned generation and interactive audio-text dialogue, CALM-based models achieve state-of-the-art performance across tasks and modalities.
• Efficient Transfer and Representation Learning: Codified modeling yields representations that generalize across tasks (tagging, genre, key, emotion) and domains (speech, music, environmental sound), with significant improvements over tag-based or continuous methods (Castellon et al., 2021, Sachidananda et al., 2022).
• Scalable Multimodal Integration: Joint modeling of audio and text tokens, with hierarchical or factorized tokenization, allows seamless integration with existing LLMs and efficient transfer between modalities (Yang et al., 4 Feb 2026).

A plausible implication is that as CALM methods mature and scale, they will form the core of general-purpose audio foundation models, supporting end-to-end audio understanding, synthesis, and multimodal reasoning in a fashion analogous to the role of LLMs in NLP, with impact across music information retrieval, speech analytics, audio captioning, and interactive AI systems.