Memorization or Interpolation ? Detecting LLM Memorization through Input Perturbation Analysis

Abstract: While LLMs achieve remarkable performance through training on massive datasets, they can exhibit concerning behaviors such as verbatim reproduction of training data rather than true generalization. This memorization phenomenon raises significant concerns about data privacy, intellectual property rights, and the reliability of model evaluations. This paper introduces PEARL, a novel approach for detecting memorization in LLMs. PEARL assesses how sensitive an LLM's performance is to input perturbations, enabling memorization detection without requiring access to the model's internals. We investigate how input perturbations affect the consistency of outputs, enabling us to distinguish between true generalization and memorization. Our findings, following extensive experiments on the Pythia open model, provide a robust framework for identifying when the model simply regurgitates learned information. Applied on the GPT 4o models, the PEARL framework not only identified cases of memorization of classic texts from the Bible or common code from HumanEval but also demonstrated that it can provide supporting evidence that some data, such as from the New York Times news articles, were likely part of the training data of a given model.

Memorization or Interpolation? Detecting LLM Memorization through Input Perturbation Analysis

The paper titled "Memorization or Interpolation? Detecting LLM Memorization through Input Perturbation Analysis" introduces a novel framework, labeled PEARL, for detecting memorization in large language models (LLMs). This research addresses critical concerns in AI regarding data privacy, intellectual property rights, and model reliability by providing a structured approach to identifying instances where LLMs produce verbatim content from their training datasets instead of generalizing from learned patterns. The underlying hypothesis of the study is termed the Perturbation Sensitivity Hypothesis (PSH), which postulates that memorized data points cause significant sensitivity in model outputs when small perturbations are applied to inputs. The introduction of PEARL represents a methodological shift from traditional, often complex methods of memorization detection, providing a black-box approach that does not necessitate access to internal model parameters or training datasets.

Core Contributions

1. Perturbation Sensitivity Hypothesis (PSH): The PSH posits that memorized content exhibits high sensitivity to input perturbations. This hypothesis is systematically applied to differentiate between memorization and interpolation in LLMs.

2. PEARL Framework: PEARL operationalizes PSH by analyzing outputs from models subjected to perturbed inputs, quantifying sensitivity, and identifying memorized data using a task-specific performance metric. This identification process enables determining whether model outputs are based on memorized data or generalized from learning.

3. Robust Assessment Across Model Types: The authors validate their hypothesis using the open-source model Pythia, with transparent training data, and the closed-source GPT-4o model, demonstrating PEARL's applicability across different domains, including code generation with HumanEval and textual data like the Bible and New York Times articles.

Experimental Validation

The authors utilize PEARL to assess memorization in LLMs across various models and datasets, revealing nuanced insights into how LLMs handle training data. In the controlled environment with Pythia, PEARL effectively distinguishes between data in the training set (The Pile) and outside it (RefinedWeb), indicating its capability to detect memorized content reliably. Furthermore, experiments reveal the impact of model size, with larger models exhibiting higher tendencies towards memorization. Applying PEARL in real-world scenarios with GPT-4o, the framework identifies notable memorization occurrences in datasets suspected to be part of training, such as HumanEval, and provides case studies for potential proprietary data usage, like New York Times articles.

Implications and Future Directions

The implications of PEARL span both practical and theoretical domains. Practically, the framework equips researchers and practitioners with a tool to detect memorization risks, contributing to AI transparency and addressing data privacy concerns. Theoretically, the PSH provides a foundation for further exploration into memorization mechanisms in LLMs and their relationships with generalization capacities. The paper invites speculation on future enhancements in AI model evaluations and their ethical data usage considerations, advocating for open science and responsible AI development.

In conclusion, this paper provides significant insights into the detection and analysis of memorization in LLMs, challenging prevailing paradigms with its innovative perturbation sensitivity approach. As AI models continue to expand, understanding memorization dynamics will be pivotal in ensuring model reliability and maintaining ethical standards in data usage. PEARL opens new avenues for introspective evaluation of AI systems, fostering developments toward more transparent and trustworthy machine learning approaches.