Augmenting Bias Detection in LLMs Using Topological Data Analysis

Abstract: Recently, many bias detection methods have been proposed to determine the level of bias a LLM captures. However, tests to identify which parts of a LLM are responsible for bias towards specific groups remain underdeveloped. In this study, we present a method using topological data analysis to identify which heads in GPT-2 contribute to the misrepresentation of identity groups present in the StereoSet dataset. We find that biases for particular categories, such as gender or profession, are concentrated in attention heads that act as hot spots. The metric we propose can also be used to determine which heads capture bias for a specific group within a bias category, and future work could extend this method to help de-bias LLMs.