Listening Between the Lines: Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection

Abstract: Early detection of dementia through speech analysis offers a non-invasive screening alternative, but capturing both acoustic and linguistic biomarkers remains challenging. We propose a multimodal framework leveraging Whisper for dual-purpose extraction: acoustic representations from encoder outputs and transcripts via automatic speech recognition (ASR). For the acoustic pathway, temporal networks with attention pooling aggregate variable-length sequences into fixed-dimensional embeddings. For the linguistic pathway, we prompt a LLM to extract interpretable features spanning lexical diversity, syntactic complexity, semantic coherence, and discourse patterns. A gated fusion network integrates both modalities. On ADReSS and ADReSSo, our method achieves F1-scores of 89.47% and 90.14%, demonstrating effective integration of acoustic and LLM-augmented linguistic features. Ablation shows that multimodal fusion consistently outperforms either modality alone.

• The paper presents a novel multimodal approach integrating ASR acoustic embeddings and LLM-extracted linguistic features to improve dementia detection accuracy.
• It employs a gated fusion network to combine frame-level Whisper embeddings with GPT-5.2 derived discourse features, achieving F1 scores up to 90.14%.
• Results and ablation studies demonstrate that incorporating statistically non-significant linguistic features can enhance diagnostic interpretability and overall performance.

Joint Learning of ASR Embeddings and LLM-Augmented Linguistics for Dementia Detection

Introduction

This work presents a comprehensive multimodal framework for the detection of dementia, specifically Alzheimer's disease (AD), through the fusion of acoustic and linguistic biomarkers derived from patients' speech. Existing approaches predominantly isolate either the "how" (acoustic signature) or the "what" (linguistic content) of patient utterances. By integrating both, the proposed method seeks to enhance both diagnostic accuracy and interpretability, leveraging state-of-the-art automatic speech recognition (ASR) and prompting strategies for LLMs.

Multimodal Architecture

The architecture comprises two principal components: an acoustic pathway using encoder-level ASR embeddings and a linguistic pathway built from LLM-augmented, interpretable discourse features. Whisper (large-v3) is employed for both ASR and as a provider of frame-level audio embeddings. The core multimodal integration is achieved through a gated fusion network, enabling learned, sample-adaptive weighting of the two modalities.

Figure 1: Overview of the proposed multimodal framework. The acoustic pathway extracts attention-pooled representations from the Whisper encoder; LLM-based linguistic features are derived from the ASR output and both modalities are adaptively fused by a gated network for classification.

Acoustic Pathway

Frame-level audio features from the Whisper encoder are processed with a temporal network (either CNN or bidirectional LSTM), followed by attention pooling. The attention mechanism focuses on diagnostically salient temporal segments, addressing the challenge of variable-length sequences. The resultant feature vector is then projected via a feed-forward architecture with LayerNorm.

Linguistic Pathway

For linguistic analysis, GPT-5.2 is prompted with ASR transcripts and structured instructions to extract a set of 46 interpretable discourse features, covering lexical diversity, syntactic complexity, semantic coherence, and meta-discourse. Feature selection reduces the set to a 29-dimensional vector optimized for downstream performance—preserving complementary features that may be statistically non-significant in univariate tests but crucial in multivariate models.

Fusion Network

A gated multimodal unit dynamically weighs each modality, enabling sample-specific integration based on relative informativeness. The fusion gate applies a sigmoid-activated linear transformation to the concatenated acoustic and linguistic vectors, followed by an element-wise blending. The fused representation is then classified into AD or cognitively normal (CN) categories.

Linguistic Feature Extraction Process

A notable methodological contribution is the joint construction of a hierarchical topic taxonomy for the Cookie Theft picture description task using LLM prompting. Unlike prior manual information unit (IU) lists, this taxonomy organizes utterances into attentional clusters reflecting spatial, thematic, and agent-centered grouping.

The prompt supports unified, sentence-level annotation along five axes: topic cluster, classification confidence, language quality (modeled after the BDAE profile), content integration, and sentence-level semantic distance. This holistic annotation is carried out in one pass per utterance, improving efficiency and consistency.

Experimental Setup

Experiments are conducted on the ADReSS and ADReSSo datasets, both based on DementiaBank’s Pitt Corpus. The datasets comprise transcribed and annotated speech associated with the Cookie Theft task, balanced (ADReSS) and unbalanced (ADReSSo) for demographics and diagnosis. Audio is processed with Whisper; transcripts receive LLM-based feature extraction. Multimodal models are trained end-to-end with AdamW optimization and early stopping.

Results and Ablations

On ADReSS and ADReSSo, the proposed model attains F1-scores of 89.47% and 90.14% respectively, outperforming all compared baselines by significant margins (up to +19.3% F1 on ADReSS). The method demonstrates high AD precision (100%) on ADReSS and balanced precision-recall on ADReSSo.

Ablation studies reveal:

• Multimodal fusion consistently outperforms unimodal models. The performance gain is especially marked over the linguistic-only baseline.
• Inclusion of statistically non-significant features enhances classification—the "optimized" 29-feature subset (where only 44.8% show $p<0.05$ in univariate testing) yields superior F1 to subsets with only significant features, highlighting indispensable multivariate interactions.
• The use of LSTM over CNN for temporal modeling is most beneficial when segment duration is unconstrained (i.e., ADReSSo), reflecting LSTM's advantage in modeling long-range dependencies.
• Discourse features highly indicative of AD include increased meta-discourse signals (C8 cluster: filled pauses, uncertainty) and reduced referential coverage.

Implications and Future Directions

The empirical findings support a paradigm where interpretable, LLM-extracted features capture linguistic impairments with a granularity and flexibility unattainable by traditional, hand-curated IU lists. The adoption of a gating-based multimodal fusion emphasizes the complementary nature of prosodic and discourse-based biomarkers—neither alone suffices for optimum diagnostic power.

From a practical standpoint, this framework establishes a strong methodological case for integrating ASR encoder-level features with LLM-driven clinical feature engineering. The reliance on cloud-based LLM APIs remains a deployment challenge in clinical contexts, especially for resource-limited or privacy-sensitive scenarios.

Theoretically, the evidence against exclusive reliance on univariate statistical significance for feature selection is consequential for the design of interpretable, multivariate clinical classifiers.

Looking forward, directions for extension include:

• Developing lightweight, open-source LLMs suitable for local inference;
• Applying the paradigm to other languages and spontaneous speech tasks;
• Conducting longitudinal modeling for tracking disease progression.

Conclusion

This work delivers a robust and interpretable multimodal framework for dementia detection that unifies ASR-based acoustic embeddings and LLM-augmented linguistic features. The demonstrated performance gains, ablation-based insights into feature interaction, and interpretable linguistic taxonomy position this approach at the forefront of computational biomarkers for cognitive decline. Future research should address deployment efficiency and generalization beyond the narrowly constrained Cookie Theft paradigm.