Context-Aware Membership Inference Attacks against Pre-trained Large Language Models

Abstract: Prior Membership Inference Attacks (MIAs) on pre-trained LLMs, adapted from classification model attacks, fail due to ignoring the generative process of LLMs across token sequences. In this paper, we present a novel attack that adapts MIA statistical tests to the perplexity dynamics of subsequences within a data point. Our method significantly outperforms prior loss-based approaches, revealing context-dependent memorization patterns in pre-trained LLMs.