Knowledge Graph-Aided ASR Error Correction

• Knowledge graph-aided ASR error correction leverages structured world knowledge to disambiguate errors and reduce word error rates in specialized domains.
• It integrates semantic, phonetic, and entity-specific features using log-linear models, embedding techniques, and LLM prompt engineering for effective rescoring and correction.
• Empirical results demonstrate significant improvements in QA accuracy and WER reduction, while challenges in scalability and domain adaptation guide future research.

Knowledge graph-aided ASR error correction refers to a class of approaches that leverage structured, graph-based representations of domain or world knowledge to improve the accuracy of automatic speech recognition (ASR) systems, particularly in domains where named entities and terminological precision are crucial. By integrating semantic, relational, and sometimes phonetic information from knowledge graphs (KGs), such systems can disambiguate entity mentions, correct misleading outputs, and reduce error rates in downstream tasks such as spoken question answering (SQA), virtual assistance, and general transcription.

1. Motivation and Problem Formulation

Conventional ASR systems frequently struggle with the recognition of rare or ambiguous entities, technical terms, and homophones, resulting in high word error rates (WER) and sentence error rates (SER), especially in domain-specific applications such as medical QA and virtual assistants. These errors often stem from acoustic confusability, limited vocabulary coverage, or insufficient linguistic context in n-gram or neural LLMs. Knowledge graph-aided frameworks address these challenges by injecting structured factual and relational knowledge—entity types, aliases, relationships, and (in some cases) phonetic confusables—into the error correction pipeline, usually at the post-ASR or lattice rescoring stage. The objective is to align ASR hypotheses more closely with real-world entity relationships or domain constraints and to exploit external world knowledge unattainable to base ASR models (Saebi et al., 2021, Kumar et al., 2017, Song et al., 1 Feb 2026).

2. Construction and Utilization of Knowledge Graphs

Effective application of KGs to ASR error correction depends on the precise construction and real-time exploitation of suitable graphs:

• Entity and Relation Extraction: In domain-agnostic settings, entities are extracted from ASR hypotheses using mention detection and linking systems (e.g., DBpedia Spotlight for TED-Lium) (Kumar et al., 2017). In medical domains, entity recognition leverages lexicons or NER models to identify domain concepts.
• Graph Construction:
  • Semantic edges: In MedSpeak, nodes represent UMLS medical concepts with standardized CUIs, linked by relations such as classifies, due_to, or plays_role (Song et al., 1 Feb 2026).
  • Phonetic edges: Unique to MedSpeak, undirected edges connect node pairs with similar Double Metaphone encodings and low Levenshtein distance in CMU-dictionary pronunciations, explicitly modeling phonetic confusability prevalent in medical speech (Song et al., 1 Feb 2026).
  • General domains: Graphs are constructed by fetching triples from resources such as DBpedia, using entity URIs and relations extracted with SPARQL queries and filtering high-degree hubs (Kumar et al., 2017).
• Representation for Inference: The retrieved KG subgraphs for all detected entities are serialized as plain text (e.g., for LLM prompting) or embedded numerically (e.g., with TransE embeddings), depending on the downstream error correction method.

3. Integration Mechanisms: Semantic and Phonetic Context in Scoring and Correction

Three principal integration methodologies are apparent across recent systems:

• Log-linear Feature Integration: In discriminative n-gram language modeling (DEAL), KG-based features (e.g., entity–entity co-occurrence, type-specific n-grams) are activated during candidate scoring, with weights learned discriminatively (Saebi et al., 2021). The overall score for a candidate hypothesis $h$ in utterance $u$ is

$s(u,h) = \sum_{f=0}^{F} w_f x_{u,h,f}$

where $f=0$ is the base system score, and $f \ge 1$ enumerates KG-induced features.

• Embedding-based Relatedness: KG triples retrieved for a hypothesis are embedded via TransE; the semantic relatedness cost between consecutive entities is computed as

$$\beta(H) = \sum_{t=1}^{T-1} \delta_t = \sum_{t=1}^{T-1} \min_n \|M_{t+1}^n - M_t^n\|$$

where $M_t^n$ is the $n$-th triple embedding for entity $e_t$ (Kumar et al., 2017).

• Prompt-based Semantic/Phonetic Context for LLMs: In MedSpeak, the union of a term's semantic and phonetic neighbors is supplied to the LLM via prompt engineering, enabling the model to resolve both context and potential confusables during correction. No explicit embedding learning is performed; instead, raw relational and phonetic context is input to the LLM (Song et al., 1 Feb 2026).

In an Editor's term for this approach, one can describe it as "contextual candidate augmentation": candidate entity corrections are presented as explicitly contextualized options for the LLM's attention mechanism.

4. Correction Algorithms and Model Architectures

The concrete error correction workflow varies by system:

• N-best/Lattice Rescoring: Both (Kumar et al., 2017) and (Saebi et al., 2021) rerank the N-best or lattice hypotheses output by a decoder using KG-derived features or cost terms. Hypotheses are rescored in a log-linear framework, with KG features (embedding distances, n-gram triggers) interpolated with acoustic and LLM scores. Feature weights are tuned by grid search or discriminative loss minimization (e.g., minimum WER objective).
• LLM-based Correction: MedSpeak employs a fine-tuned Llama-3.1-8B-Instruct LLM. The model is trained to take as input the noisy ASR transcript, multiple-choice options, and truncated KG context—outputting both a corrected transcript and the predicted answer. Formally:

$$L(\theta) = -\sum_{i=1}^T \log P_\theta(y_i \mid x, y_{<i})$$

where the KG context $x$ comprises all semantic and phonetic neighbors for detected terms, serialized to a capped length (Song et al., 1 Feb 2026).

In inference, the LLM processes the prompt:

SYSTEM_MSG User(transcript=ŷ, options=Options, KG=KG_ctx)

and outputs:

Corrected Text: ⊲ 𝓉̃ Correct Option: ⊲ o*

5. Empirical Results and Domain-Specific Performance

Empirical studies consistently demonstrate the value of KG-augmented correction, particularly in entity-rich or technical domains:

Model	MedSpeak (Medical SQA)	DEAL2-rpc (VA, tail entities)	KG-Rescore (TED-Lium)
Baseline Accuracy	50.2–83.7%	18.26–78.43% SER (city/music/soccer)	18.2% WER
With KG Integration	93.4% (QA Acc.), 29.9% WER	10.59–62.85% SER	17.9% WER

• MedSpeak: On three spoken medical QA datasets (overall ~47 hours), the inclusion of medical KG context reduces WER from 35.8% to 29.9% and improves QA accuracy from 83.7% to 93.4%. Gains are most pronounced in specialties prone to phonetic confusion (e.g., virology, anatomy); ablation studies show that KG context yields 5–7 WER point improvements beyond LLM fine-tuning alone (Song et al., 1 Feb 2026).
• Entity-Aware LLMs: In tail-entity-heavy virtual assistant tasks, KG-aware discriminative LMs reduce SER by 20–40% for rare entities with negligible impact (+0.3% SER) on general traffic (Saebi et al., 2021).
• General ASR: KG rescoring improves WER modestly (from 18.2% to 17.9% on TED-Lium), with the greatest benefit observed in hypotheses rich in well-linked entities (Kumar et al., 2017).

6. Limitations and Open Challenges

Despite demonstrated improvements, knowledge graph-aided ASR error correction faces several persistent challenges:

• Entity Coverage and Linking: Out-of-vocabulary or mislinked entities in the KG cannot be corrected or leveraged. Robust entity linking remains crucial for both semantic embedding and feature-based systems (Kumar et al., 2017, Saebi et al., 2021).
• Scalability and Latency: Real-time retrieval and embedding of large subgraphs (minimum hundreds of triples per utterance in some workflows) can introduce inference latency, especially for dynamic SPARQL-based approaches (Kumar et al., 2017).
• Domain Adaptation: The effectiveness of KGs is highly domain-dependent. General graphs like DBpedia exhibit limited coverage or precision in specialized domains, while construction and maintenance of domain-specific KGs (e.g., UMLS) require substantial curation (Song et al., 1 Feb 2026).
• Integration Depth: Most systems operate in a reranking or post-processing mode, influencing only the N-best/lattice selection or LLM outputs; tighter integration earlier in the decoding process could potentially yield greater improvements (Saebi et al., 2021).
• Expansion to Neural and Multilingual Models: Current approaches show limited direct neural integration (outside of prompt-based LLMs, e.g., MedSpeak), and porting to languages other than English requires high-quality, multilingual KGs and phonetic models.

A plausible implication is that future work will likely focus on end-to-end neural approaches that fuse KG knowledge at the representation level, jointly optimize over both acoustic and symbolic knowledge, and generalize across languages and domains.

7. Summary and Outlook

Knowledge graph-aided ASR error correction represents a convergence of symbolic AI, statistical/n-gram modeling, and deep learning. Foundational frameworks—including entity-aware LLMs, embedding-based rescoring, and LLM prompt engineering—demonstrate consistent, if sometimes moderate, reductions in error rate, particularly for proper name, terminological, and entity-centric utterances. In clinical spoken QA, MedSpeak establishes a state-of-the-art paradigm by marrying UMLS-derived semantic and phonetic structure with modern LLMs in a two-step pipeline—retrieval and fine-tuning—achieving both superior transcription correction and QA performance (Song et al., 1 Feb 2026). The trajectory of the field points toward greater synergy between explicit, interpretable KGs and the generalization capacity of neural models, with ongoing challenges in scalability, coverage, and cross-linguistic adaptation (Kumar et al., 2017, Saebi et al., 2021).