Exploring Neural Net Augmentation to BERT for Question Answering on SQUAD 2.0

Abstract: Enhancing machine capabilities to answer questions has been a topic of considerable focus in recent years of NLP research. LLMs like Embeddings from LLMs (ELMo)[1] and Bidirectional Encoder Representations from Transformers (BERT) [2] have been very successful in developing general purpose LLMs that can be optimized for a large number of downstream language tasks. In this work, we focused on augmenting the pre-trained BERT LLM with different output neural net architectures and compared their performance on question answering task posed by the Stanford Question Answering Dataset 2.0 (SQUAD 2.0) [3]. Additionally, we also fine-tuned the pre-trained BERT model parameters to demonstrate its effectiveness in adapting to specialized language tasks. Our best output network, is the contextualized CNN that performs on both the unanswerable and answerable question answering tasks with F1 scores of 75.32 and 64.85 respectively.