Behavior Gated Language Models

Abstract: Most current language modeling techniques only exploit co-occurrence, semantic and syntactic information from the sequence of words. However, a range of information such as the state of the speaker and dynamics of the interaction might be useful. In this work we derive motivation from psycholinguistics and propose the addition of behavioral information into the context of language modeling. We propose the augmentation of LLMs with an additional module which analyzes the behavioral state of the current context. This behavioral information is used to gate the outputs of the LLM before the final word prediction output. We show that the addition of behavioral context in LLMs achieves lower perplexities on behavior-rich datasets. We also confirm the validity of the proposed models on a variety of model architectures and improve on previous state-of-the-art models with generic domain Penn Treebank Corpus.