SALLM: Speech-Aware LLMs

• SALLM is a class of systems that jointly process acoustic signals and text to enable tasks like transcription, translation, and spoken question answering.
• Innovative architectures, including acoustic tokenizers and crossmodal adapters, facilitate seamless fusion between speech and language modalities.
• Reinforcement learning methods and benchmarking frameworks validate SALLM performance while highlighting challenges in long utterance and multi-turn dialogue handling.

Speech-Aware LLMs (SALLM) represent a class of neural architectures and training methodologies that enable LLMs to jointly process, understand, and generate language conditioned on speech or rich acoustic inputs. As of 2026, SALLM encompasses both general-purpose speech-LLMs covering transcription, translation, open-ended spoken question answering, and conversational audio understanding, as well as frameworks for robust evaluation and secure deployment. The field integrates advances in encoder architectures, crossmodal fusion, reinforcement learning fine-tuning, and application-driven benchmarks targeting real-world multimodal scenarios.

1. SALLM: Definition, Scope, and Historical Context

SALLM refers to Speech-Aware LLMs—LLMs or closely coupled multi-module systems specifically architected and trained to reason over both text and acoustic signals. The impetus for SALLM arose from limitations in classical sequence-to-sequence ASR (automatic speech recognition), which focused purely on transcription and neglected downstream reasoning, and from the inadequacy of standard LLMs to natively parse acoustic or speech context. Early systems such as “SALM: Speech-augmented LLM” (Chen et al., 2023) demonstrated that integrating a speech front-end (Fast Conformer) with frozen, instruction-tuned LLMs enabled competitive multitask ASR and speech translation while also supporting in-context learning for speech-driven keyword biasing.

Later architectural advances, such as the inclusion of acoustic tokenizers (Q-Former), crossmodal adapters (e.g., LoRA injected into LLM layers), and modular loss functions, have broadened SALLM applications to spoken question answering, audio event reasoning, multimodal captioning, and robust dialogic AI that can operate under challenging conditions including overlapping speech and noise (Ao et al., 19 Mar 2025, Elmakies et al., 21 Sep 2025).

2. Architectures and Training Paradigms

SALLM architectures are typically composed of:

• Speech/Audio Encoder: Converts raw waveform $x$ to temporally downsampled acoustic embeddings, often using conformer or CTC models (Chen et al., 2023, Ao et al., 19 Mar 2025, Elmakies et al., 21 Sep 2025).
• Modality Adapter or Q-Former: Connects the time-dense audio encoder output to the LLM token space, via windowed attention, downsampling, and linear projections (Elmakies et al., 21 Sep 2025).
• Frozen or Lightly-tuned LLM Backbone: A transformer-based LLM (e.g., InternLM2-chat-7B, Megatron-LM) that consumes acoustic tokens (as “prefix” or “soft” tokens) concatenated with text prompt tokens and optional in-context exemplars (Chen et al., 2023, Ao et al., 19 Mar 2025).
• Crossmodal Fusion: Prefix-tuning, learned cross-attention, or concatenation to blend acoustic and text representations (Ao et al., 19 Mar 2025).
• Output Head: For next-token prediction (ASR, translation) and/or discrete task heads (classification, reasoning).

Training objectives vary by application. For core speech tasks, SALLMs minimize cross-entropy for ASR, AST (speech translation), and task-specific outputs. To bridge the speech-text modality gap, “speech supervised in-context training” augments training with synthetic keyword/context prompts to induce zero-shot in-context behavior (Chen et al., 2023). RL-based fine-tuning, notably Group Relative Policy Optimization (GRPO) with BLEU-based reward, has been shown to outperform supervised fine-tuning, especially for open-format tasks in spoken question answering and translation (Elmakies et al., 21 Sep 2025).

3. Supervised and Reinforcement Learning for SALLM

Supervised Instruction Tuning operates over large parallel datasets—LibriSpeech for ASR, IWSLT or CoVoST for speech translation—with multitask optimization:

$$L_{\mathrm{total}} = \lambda_{ASR}\,L_{ASR} + \lambda_{AST}\,L_{AST} + \lambda_{ICT}\,L_{ICT}$$

where $L_{ICT}$ is the speech supervised in-context training loss, explicitly encouraging keyword or context prompt following (Chen et al., 2023).

Reinforcement Learning for SALLM: GRPO is an on-policy RL algorithm adapted for SALLM optimization in open-ended tasks (Elmakies et al., 21 Sep 2025). For each prompt, $G$ completions are sampled, BLEU rewards computed, normalized, and used in a PPO-style clipped loss with additional KL regularization:

$$\mathcal{L}_{\mathrm{DAPO}}(\theta) = -\frac{1}{\sum_i |o_i|}\sum_{i=1}^G \sum_{t=1}^{|o_i|} \ell_{i,t} + \beta D_{KL}(\pi_\theta \Vert \pi_{\mathrm{ref}})$$

Advantages are normalized within each group, and off-policy extensions (MP-GRPO) inject reference samples into the reward grouping. GRPO with BLEU reward consistently outperforms SFT baselines for both SQA and AST; e.g., on LibriSQA, Granite8B+GRPO achieves a BLEU of 46.40 (vs. 42.34 for SFT), and on CoVoST2 En→De, Granite8B+GRPO reaches BLEU 35.08 (vs. 31.62 for SFT) (Elmakies et al., 21 Sep 2025).

4. Benchmarking, Evaluation, and Analysis Frameworks

SALLM benchmarks and evaluation proceed along several axes:

• SA-Eval Benchmark: For assessment of speech-instructed, context-aware audio understanding, SA-Eval covers audio event classification, captioning, and QA across VGGSound, AudioSet, FSD50K, AudioCaps, Clotho, and Clotho-AQA (Ao et al., 19 Mar 2025). Metrics include accuracy, macro-F1, CIDEr, SPICE, SPIDEr, and QA accuracy. Solla SALLM matches or outperforms speech LLM baselines, especially in overlapping/noisy conditions (≥99% instruction following accuracy in “hard” mode).
• Keyword Biasing and In-context Learning: Specialized evaluation on ASR keyword recall shows in-context prompt biasing in SALM matches classical shallow fusion without dedicated bias architectures (Chen et al., 2023).
• Open-Format Spoken QA and Translation: RL-based evaluation uses BLEU, BERTScore, ROUGE, and METEOR. Empirical results demonstrate consistent improvements of GRPO-SALLMs over supervised-only training across all metrics (Elmakies et al., 21 Sep 2025).
• Limitations in Reasoning Scope: Single-turn dialogue, text-only output, and limited focus on fine-grained acoustic scene parsing remain as current boundaries in SALLM evaluation (Ao et al., 19 Mar 2025).

5. Strengths, Weaknesses, and Ablation Insights

Strengths:

• Unified multitask architecture enables simultaneous ASR, AST, keyword-boosted recognition, and spoken QA (Chen et al., 2023, Ao et al., 19 Mar 2025, Elmakies et al., 21 Sep 2025).
• RL fine-tuning (GRPO with BLEU) robustly optimizes for open-format answers in spoken QA and translation, outperforming SFT for core generation metrics (Elmakies et al., 21 Sep 2025).
• Modular architectures (LoRA adapters, Q-Formers, AT modules) facilitate model scaling and domain adaptation.
• In-context learning via supervised context significantly enhances performance in keyword-sensitive applications.

Limitations:

• Modality bottleneck: SALLMs typically emit only text, with no native speech or audio generation (Ao et al., 19 Mar 2025).
• Performance degrades for long utterances or as the number of biasing keywords increases (due to false-accept rate) (Chen et al., 2023).
• Generalization to multi-turn conversational contexts and multi-source (e.g., multi-mic, spatial) audio remains unaddressed or at an early stage.

Ablation Findings:

• Adapter depth: two conformer layers optimal for matching speech and text rates; further layers yield marginal gain (Chen et al., 2023).
• LoRA rank: higher rank stabilizes fine-tuning.
• GRPO group size, KL penalty, and reward function (BLEU > ROUGE/METEOR) are critical for balancing score and variance in RL training (Elmakies et al., 21 Sep 2025).
• Removal of AT modules or ASR-assisted loss in speech QA models degrades instruction-following and classification performance (Ao et al., 19 Mar 2025).

6. Comparison to Related Multi-Modal and Audio-LLMs

Compared to models focused solely on spatial audio (e.g., “SALM: Spatial Audio LLM” (Hu et al., 22 Jul 2025)), SALLMs are optimized for joint processing of speech instruction and non-speech audio, handling real-world mixtures and questions. While spatial audio models address the alignment of semantic and directional cues via contrastive learning, SALLMs emphasize robust crossmodal fusion for speech understanding, dynamic instruction following under noise, and reinforcement learning from task-specific rewards.

Compared to security-oriented frameworks, e.g., “SALLM: Security Assessment of LLM-Generated Code” (Siddiq et al., 2023), which focus on benchmarking LLM code generation for vulnerabilities, SALLM in the speech context is a model/system class rather than an evaluation pipeline. However, both bring rigor and new methodologies to their target domains—SALLM for robust, auditory-aware language modeling, and SALLM (security) for secure code generation assessment.

7. Future Directions

Active research frontiers in SALLM include:

• Conversational Audio Modeling: Extending SALLMs to multi-turn dialogue with memory and reasoning across utterance history (Ao et al., 19 Mar 2025).
• Audio Generation: Coupling speech encoders with neural codec generators to yield spoken or audio-conditioned textual output.
• Multi-source and Spatial Audio Processing: Architectures generalizing to spatial, multichannel, or complex auditory scenes, potentially integrating principles from spatial audio language modeling (Hu et al., 22 Jul 2025).
• Crossmodal Curricula and Augmented Supervision: Combining GRPO-based RL with curriculum-style group sizing, multiple reference integration, and neural/composite reward functions (BLEU+BERTScore).
• Scaling and Domain Adaptation: Porting SALLMs to larger parameter regimes and low-resource linguistic or acoustic domains with efficient adaptation or alignment modules.
• Deeper Crossmodal Fusion: Learning more granular fusion points beyond initial prefixing or windowed attention, such as multi-level cross-attention between speech and language at all LLM layers.

A plausible implication is that as new architectures and benchmarks emerge, SALLM will increasingly undergird the next generation of multi-modal, conversational, and perceptually rich AI agents able to reason fluently across both spoken and written modalities (Elmakies et al., 21 Sep 2025, Ao et al., 19 Mar 2025, Chen et al., 2023).