External Language Model Integration for Factorized Neural Transducers

Abstract: We propose an adaptation method for factorized neural transducers (FNT) with external LLMs. We demonstrate that both neural and n-gram external LMs add significantly more value when linearly interpolated with predictor output compared to shallow fusion, thus confirming that FNT forces the predictor to act like regular LLMs. Further, we propose a method to integrate class-based n-gram LLMs into FNT framework resulting in accuracy gains similar to a hybrid setup. We show average gains of 18% WERR with lexical adaptation across various scenarios and additive gains of up to 60% WERR in one entity-rich scenario through a combination of class-based n-gram and neural LMs.