Interactively Providing Explanations for Transformer Language Models

Abstract: Transformer LLMs are state of the art in a multitude of NLP tasks. Despite these successes, their opaqueness remains problematic. Recent methods aiming to provide interpretability and explainability to black-box models primarily focus on post-hoc explanations of (sometimes spurious) input-output correlations. Instead, we emphasize using prototype networks directly incorporated into the model architecture and hence explain the reasoning process behind the network's decisions. Our architecture performs on par with several LLMs and, moreover, enables learning from user interactions. This not only offers a better understanding of LLMs but uses human capabilities to incorporate knowledge outside of the rigid range of purely data-driven approaches.