Removing Spurious Correlation from Neural Network Interpretations

Abstract: The existing algorithms for identification of neurons responsible for undesired and harmful behaviors do not consider the effects of confounders such as topic of the conversation. In this work, we show that confounders can create spurious correlations and propose a new causal mediation approach that controls the impact of the topic. In experiments with two LLMs, we study the localization hypothesis and show that adjusting for the effect of conversation topic, toxicity becomes less localized.