Sparse associative memory based on contextual code learning for disambiguating word senses

Abstract: In recent literature, contextual pretrained LLMs (LMs) demonstrated their potential in generalizing the knowledge to several NLP tasks including supervised Word Sense Disambiguation (WSD), a challenging problem in the field of Natural Language Understanding (NLU). However, word representations from these models are still very dense, costly in terms of memory footprint, as well as minimally interpretable. In order to address such issues, we propose a new supervised biologically inspired technique for transferring large pre-trained LLM representations into a compressed representation, for the case of WSD. Our produced representation contributes to increase the general interpretability of the framework and to decrease memory footprint, while enhancing performance.