LLM-Based Automatic Speech Recognition

• LLM-ASR is a paradigm integrating speech encoders and large language models to transcribe speech with enhanced accuracy and context-awareness.
• It employs modular pipelines—like encoder–adapter–decoder and soft discretization—to align acoustic features with token representations, boosting multi-dialect and rare word recognition.
• Advanced training and optimization methods, including self-supervised pretraining, adapter tuning, and reinforcement learning, reduce compute costs while improving performance.

LLM-based Automatic Speech Recognition (ASR) constitutes an architectural paradigm in which LLMs are integrated with speech encoders to transcribe spoken utterances into text. This paradigm leverages the contextual modeling, few-shot generalization, and instruction-following capabilities of contemporary LLMs to address transcription accuracy, context-aware decoding, rare word recognition, multi-dialect robustness, and domain adaptation. The fundamental workflow interposes a speech encoder, adapter module(s), and a decoder-only LLM, which together support end-to-end and modular systems, as well as hybrids with iterative and retrieval-augmented recognition.

1. Model Architectures and Modality Bridging

LLM-ASR systems universally employ a pipeline in which the audio waveform is encoded into high-dimensional feature sequences, subsequently projected or discretized to align with the LLM's input embedding space. Canonical architectures include:

• Encoder–Adapter–Decoder (EAD): A high-capacity speech encoder (e.g., Whisper, Conformer, HuBERT) produces framewise continuous embeddings; a lightweight adapter (e.g., a two-layer MLP, Transformer block, MoE connector) down-samples and linearly projects these embeddings into the LLM’s token space (Wang, 22 Feb 2025, Yang et al., 6 Jun 2025, Geng et al., 2024, Xu et al., 2024, Bai et al., 2024).
• Soft Discretization/VQ: A vector quantization module, where encoder outputs are softly mapped, via temperature-weighted kernel or cosine similarity, onto the LLM vocabulary embedding space. This yields mixture-based representations that preserve acoustic nuance while operating within the LLM’s discrete manifold (Yang et al., 6 Jun 2025).
• Audio-conditioned LLM Fusion: Adapters and projectors concatenate audio embeddings or quantized representations with instruction/context tokens to form the full input to a frozen or LoRA-tuned decoder-only LLM (e.g., Qwen2-7B, Baichuan2-7B, LLaMA, GPT-2).

Discrete and soft discretization strategies mitigate the modality gap and facilitate alignment between encoder-produced continuous signals and the LLM’s token-only representations, yielding significant improvements on out-of-domain speech and accent/dialect robustness (Yang et al., 6 Jun 2025).

2. Training Paradigms and Optimization

State-of-the-art training for LLM-ASR systems encompasses multiple stages:

• Self-supervised pretraining of the speech encoder, often masked-frame prediction with quantized targets (a la HuBERT) over millions of unlabeled hours, followed by supervised fine-tuning on labeled ASR pairs (Bai et al., 2024).
• Adapter training: Only the adapter module is unfrozen while the encoder and LLM remain fixed to facilitate rapid alignment (Geng et al., 2024, Wang, 22 Feb 2025, Mu et al., 6 Aug 2025).
• Full integration with LoRA: Low-rank adaptation modules within the LLM are enabled to impart flexibility and avoid catastrophic forgetting of language priors (Yang et al., 6 Jun 2025, Xu et al., 2024, Mu et al., 6 Aug 2025, He et al., 31 May 2025).
• Contextual SFT: ASR pairs mixed with context triples (e.g., dialogue histories, hot-word lists, domain metadata), enforcing joint attention over both text-based and acoustic cues to elicit strong context-aware decoding (Bai et al., 2024, Song et al., 31 Dec 2025, He et al., 31 May 2025).
• Reinforcement Learning (RL): MWER/WWER or LLM-based reward optimization (RAFT, DPO, GRPO) are used to adapt ASR models for improved named entity recognition and domain customization, outperforming self-training and LLM post-processing (Ling et al., 5 Jun 2025, Bai et al., 2024).

Scaling laws for LLM-ASR detail a power-law relation between computation budget (FLOPs) and error rate; pretraining the encoder independently and freezing the LLM significantly reduces overall compute while preserving error reduction (up to 21.1% CER reduction at 49.9% less FLOPs) (Mu et al., 6 Aug 2025).

3. Contextual, Conversational, and Rare Word Recognition

LLMs provide unique capabilities in modeling long-range dependencies, contextual understanding, and rare/zero-shot word recall:

• Prompt-Conditioned Decoding: Inclusion of user/domain-provided prompts (e.g., hot-word lists, bias lists, meeting participant names) enables fine-grained contextual biasing through logit scaling and beam search bias mechanisms (Song et al., 31 Dec 2025, He et al., 31 May 2025).
• Multi-modal retrieval and selection: MARS augments conversational ASR by ranking historical utterances based on acoustic (DTW, cosine) and textual similarities, selecting a near-ideal context to maximize recognition accuracy in long-form, contextually entangled speech (Mu et al., 2 Aug 2025).
• Retrieval-Augmented Generation (LA-RAG): Token-level speech retrieval from FAISS-indexed datastores, followed by in-context prompt construction, allows the LLM to leverage previously seen acoustic patterns for accent/dialect adaptation (Li et al., 2024).
• Zero-shot Rare Word Recognition: LLM-ASR architectures confer consistent reduction in rare-word WER over conventional Zipformer-Transducer models when paired with large, high-quality pseudo-labeled datasets (e.g., YouTube, Whisper V3), with adapter placement and data quality being critical for optimal results (Wang, 22 Feb 2025).

Ablation studies confirm adapter placement, encoder robustness, and prompt integration as determinants of domain and context generalization (Geng et al., 2024, Wang, 22 Feb 2025, Bai et al., 2024, He et al., 31 May 2025).

4. Efficiency, Inference Acceleration, and Robustness

High recognition accuracy in LLM-ASR comes at the cost of inference latency due to autoregressive token-by-token generation. Recent advances address this via:

• Speculative Decoding: SpecASR implements adaptive draft generation with dynamic sequence length, draft recycling using masked decoding, and sparse token tree generation to balance latency and acceptance rates. This yields 3.04–3.79× speedup over greedy and up to 1.84× over vanilla speculative methods, without loss of recognition accuracy (Wei et al., 24 Jul 2025).
• Diffusion-based Decoding: dLLM-ASR employs prior-guided discrete diffusion to enable fast parallel sequence generation, length-adaptive pruning, and confidence-based early exit, reaching 4.44× speedup over autoregressive Whisper–LLaMA3 while maintaining competitive WER (Tian et al., 25 Jan 2026).
• Noise-enriched Training: Index-ASR combats hallucination and repetitive outputs by mixing real noisy and clean speech at training; aggressive SNR augmentation encourages acoustic grounding and curbs over-reliance on the LLM’s text prior (Song et al., 31 Dec 2025).

Prompt-based contextual customization interfaces and robust encoder pretraining further mitigate hallucination, rendering LLM-ASR applicable to low-resource, multi-dialect, and domain-specific settings (Song et al., 31 Dec 2025, He et al., 31 May 2025, Bai et al., 2024).

5. Evaluation Metrics and Semantic Assessment

Traditional Word Error Rate (WER), defined by the sum of substitutions, deletions, and insertions normalized by reference length, inadequately delineates downstream impact, especially in LLM-centric pipelines:

• Answer Error Rate (AER): Measures semantic divergence in LLM task outputs caused by ASR errors, operationalized as the proportion of question–answer pairs for which LLM outputs deviate between clean and ASR transcripts. Empirical evidence demonstrates AER substantially exceeds raw WER (often by 10–30 percentage points), and reveals that semantically critical errors dominate downstream failures despite low WER (Pulikodan et al., 22 Jul 2025).
• LLM Correction: LLM-based post-processing yields significant WER reductions mainly when input WER is high (>10%), but introduces risk of paraphrastic deviations when applied to already accurate transcripts (Pulikodan et al., 22 Jul 2025, Min et al., 2023).
• Specialized Metrics: Rare-word WER (R-WER), entity WER (EWER), and subjective intelligibility scales are used to measure system effectiveness in challenging or customized contexts (Wang, 22 Feb 2025, Ling et al., 5 Jun 2025, Bai et al., 2024).

Best practices for pipeline design mandate AER benchmarking for model selection and correction strategy validation (Pulikodan et al., 22 Jul 2025).

6. Applications, Customization, and Limitations

LLM-ASR systems have been deployed in diverse settings:

• Medical Diagnostics: Two-stage pipelines employing Whisper ASR and LoRA-adapted LLMs robustly transcribe noisy medical call recordings and map transcripts to diagnostic classes, with static equalization and augmentation enforcing microphone/ambient invariance (Kumar, 18 Feb 2025).
• Multilingual and Code-Switching ASR: Mixture-of-Experts connectors and insertion/deletion of interruption token (IDIT) mechanisms facilitate code-switching recognition by language-specific expert routing and tokenizer granularity control, reaching state-of-the-art mixed error rates (MER) with far fewer trainable parameters (Zhang et al., 2024).
• Speech Translation and Multi-task Learning: MooER achieves competitive ASR and AST (BLEU=25.2) using an adapter-plus-LoRA architecture trained on pseudo-labeled data, ablation studies validating granularity and encoder selection as key (Xu et al., 2024).
• Multi-talker and Contextual Biasing: Unified frameworks combine SOT-style overlapping speech transcription and rare-word prompting, supported by two-stage bias list filtering and prompt injection mechanisms, outperforming baseline LLM and post-processing methods (He et al., 31 May 2025).

Limitations include reliance on large parameter models (latency and cost), frozen modules (plasticity constraints), minimal support for streaming, and the need for sophisticated data filtering, adapter design, and hyperparameter tuning (Min et al., 2023, Yang et al., 6 Jun 2025, Tian et al., 25 Jan 2026, Cohen et al., 4 Aug 2025). Ongoing research targets improved phoneme modeling, efficient context retrieval, latency minimization, and robust domain adaptation.

7. Prospects and Research Directions

Active lines of research include:

• Unified Context-Aware ASR: Training on context-rich triples (e.g., text, audio, entity lists) combined with reinforcement learning and joint beam-search promises further gains for rare word and dialogue recall (Bai et al., 2024, Ling et al., 5 Jun 2025).
• Generalization to Multilingual and Low-resource Contexts: Modular architectures and retrieval-augmented strategies yield improved accuracy for dialects, accents, and code-switching with limited labeled data (Li et al., 2024, Zhang et al., 2024) [(Song et al., 2024)*].
• Efficient Scaling: Multi-stage training (encoder-first integration) and power-law compute-error scaling provide practical design guidelines for compute-constrained environments (Mu et al., 6 Aug 2025).
• Hybrid Decoding Algorithms: Iterative MAP combination of AM and LLM modules enables fully separable, zero-shot, and explainable systems, with substantial gains on complex speech and domain-specific vocabulary (Cohen et al., 4 Aug 2025).
• Semantic Fidelity Evaluation: Incorporation of AER and related measures supports richer, application-aligned assessment of ASR system fitness (Pulikodan et al., 22 Jul 2025).

A plausible implication is that, as adaptation techniques and retrieval-augmented prompts mature, LLM-ASR will subsume context-aware, domain-specific, and multi-party recognition tasks previously approached by specialized end-to-end or hybrid models.

---

*No detailed technical information extractable from (Song et al., 2024) due to lack of PDF/source, but the summary claim that LLM-ASR yields a relative gain of 12.8% over Whisper in low-resource ASR, with Whisper excelling in Mandarin-English code-switching, is noted.