Advancing LLM-based phoneme-to-grapheme for multilingual speech recognition

Abstract: Phoneme-based ASR factorizes recognition into speech-to-phoneme (S2P) and phoneme-to-grapheme (P2G), enabling cross-lingual acoustic sharing while keeping language-specific orthography in a separate module. While LLMs are promising for P2G, multilingual P2G remains challenging due to language-aware generation and severe cross-language data imbalance. We study multilingual LLM-based P2G on the ten-language CV-Lang10 benchmark. We examine robustness strategies that account for S2P uncertainty, including DANP and Simplified SKM (S-SKM). S-SKM is a Monte Carlo approximation that avoids CTC-based S2P probability weighting in P2G training. Robust training and low-resource oversampling reduce the average WER from 10.56% to 7.66%.

• The paper introduces a novel S-SKM method to efficiently handle phoneme uncertainty and noise in multilingual ASR.
• It applies robust training and low-resource oversampling to substantially reduce word error rates in low-resource languages.
• Findings indicate that unified LLM-based P2G systems nearly match traditional WFST baselines while simplifying multilingual deployment.

Multilingual Phoneme-to-Grapheme Conversion with LLMs for ASR

Introduction

Phoneme-based ASR architectures decompose the speech recognition pipeline into S2P and P2G modules, facilitating efficient multilingual and cross-lingual modeling. While recent work demonstrates that LLMs are powerful for P2G conversion, extending LLM-based P2G systems to the multilingual regime raises two key challenges: precise language-aware generation and managing severe cross-language data imbalance. This paper investigates robustness strategies for LLM-based P2G in a challenging multilingual context, introducing the Simplified Sampling-K Marginalization (S-SKM) method and data balancing approaches. Evalutions are conducted on CV-Lang10, a ten-language Common Voice subset.

Multilingual LLM-based Phoneme-to-Grapheme System

The overall system is a modular pipeline, where Whistle-large (543M parameters) is deployed for S2P, utilizing a Conformer+CTC architecture, while fine-tuned Qwen3 LLMs (1.7B and 4B) perform P2G. Input speech is first mapped to phoneme hypotheses in a language-independent IPA space. The P2G LLM then predicts the language label followed by the graphemic sequence, unifying language identification and orthographic decoding within a single model.

Multilingual P2G systems must address (i) language-aware orthography generation from universal phonemic input, and (ii) strong class imbalance due to uneven language resource distribution. The phoneme coverage in the multilingual dataset demonstrates a mixture of shared and language-specific phoneme subsets, highlighting the need for advanced representation balancing:

Figure 1: Phoneme occurrence counts across 10 languages in CV-Lang10; cells denote the training set count for each phoneme-language pair.

Robustness Strategies for Noisy S2P Hypotheses

Classical WFST-based P2G decoding is brittle to S2P errors and impractical at multilingual scale due to reliance on hand-crafted lexicons and decoding graphs. LLM-based P2G requires robustness against noisy S2P outputs. Several robustness training paradigms are evaluated:

• Data Augmentation with Noisy Phonemes (DANP): Enriches training with perturbed phoneme hypotheses (N-best lists, random samples, checkpoint-agglutinated variants), simulating realistic S2P error distributions.
• Top-K Marginalization (TKM): Trains the P2G model over top-$K$ phoneme candidates weighted by S2P probabilities, approximating the correct output marginal.
• Sampling-K Marginalization (SKM): Improves upon TKM by sampling diverse phoneme hypotheses from the S2P CTC posterior, capturing more representative S2P uncertainty, but incurs training overhead due to explicit CTC probability computations.

Simplified Sampling-K Marginalization (S-SKM)

To improve the tractability of SKM in large-scale multilingual settings, S-SKM eliminates the requirement to compute CTC-based S2P hypothesis probabilities during each P2G training iteration. By employing an equal-weighted Monte Carlo estimator over sampled CTC paths, S-SKM closely matches the effectiveness of SKM while improving efficiency, particularly beneficial when the total number of languages and utterances is substantial.

Monolingual ablation studies on Polish and German demonstrate that S-SKM achieves WER performance nearly identical to SKM, with differences within 0.15 percentage points, but significantly reduces training complexity and GPU compute per update.

Data Balancing with Low-Resource Oversampling

P2G performance on low-resource languages is substantially degraded under naive multilingual fine-tuning, attributed to update frequency disparities. The paper applies simple but highly effective low-resource language oversampling to enforce a minimum number of training updates per language, thus decoupling model quality from corpus imbalance. The target is empirically set so that every language achieves at least 240 hours of effective exposure. This straightforward mechanism is orthogonal to the robustness strategies and yields considerable gains in average and low-resource-specific WER.

Experimental Results

All strategies are assessed on CV-Lang10, considering both absolute WER and LID accuracy. Key findings include:

• Naive Multilingual Fine-tuning: Without explicit robustness strategies or balancing, LLM-P2G achieves reasonable performance on high-resource languages (e.g., English, Spanish, French, Italian), but incurs significant WER degradation on low-resource languages (e.g., Tatar at 23.25% WER, Kyrgyz at 10.07%).
• Robust Training with DANP/S-SKM: Both strategies improve WER across the board, reflecting the criticality of modeling S2P uncertainty even in multilingual settings.
• Oversampling Effect: When combined with S-SKM, low-resource oversampling reduces the unweighted mean WER from 9.64% to 7.66%. Low-resource language WERs are sharply reduced (e.g., Kyrgyz WER drops to 2.82%, Tatar to 14.34%), with no meaningful degradation on high-resource languages.
• Comparison to WFST Baseline: WFST-based P2G remains slightly superior in unweighted average WER (6.56%), especially for the lowest-resource languages, but the unified LLM-based system outperforms WFST on the training-hours-weighted average WER (8.22% vs. 9.20%). Thus, for practical deployments without language-specific lexicon engineering, unified LLM-P2G is highly competitive.

The system consistently attains high language identification accuracy across all settings (>98.7%), confirming that LID and grapheme generation can be reliably unified within the LLM-P2G pipeline.

Implications and Future Directions

This study demonstrates that strong LLM-based P2G models can be trained for multilingual ASR with minimal infrastructure. The S-SKM estimator provides an efficient training paradigm that supports system expansion to new languages without re-tuning CTC marginal weights. The persistent gap with the WFST baseline for the lowest-resource languages suggests that even more aggressive data-centric approaches or transfer strategies (e.g., joint S2P/P2G training, unsupervised or semi-supervised data utilization) may be beneficial, as well as research toward S2P architectures with improved sample efficiency for rare phoneme distributions.

End-to-end pronunciation-lexicon free models, which jointly optimize S2P and P2G, may subsume the observed bottlenecks and further close the gap with classic pipelines, as indicated by recent work on joint stochastic approximation (2603.29217).

Conclusion

Unified multilingual LLM-based P2G models, trained with robust marginalization-based objectives and data balancing, now approach or match the performance of classic WFST-based approaches for high- and mid-resource languages, while yielding substantial engineering simplification. S-SKM training ensures tractability, delivering strong LID and WER even under severe data imbalance. Future research should focus on improved S2P architectures, deeper low-resource transfer, and holistic end-to-end phoneme-intermediate modeling to further advance multilingual ASR.