Detecting the Use of Generative AI in Crowdsourced Surveys: Implications for Data Integrity

Abstract: The widespread adoption of generative AI (GenAI) has introduced new challenges in crowdsourced data collection, particularly in survey-based research. While GenAI offers powerful capabilities, its unintended use in crowdsourcing, such as generating automated survey responses, threatens the integrity of empirical research and complicates efforts to understand public opinion and behavior. In this study, we investigate and evaluate two approaches for detecting AI-generated responses in online surveys: LLM-based detection and signature-based detection. We conducted experiments across seven survey studies, comparing responses collected before 2022 with those collected after the release of ChatGPT. Our findings reveal a significant increase in AI-generated responses in the post-2022 studies, highlighting how GenAI may silently distort crowdsourced data. This work raises broader concerns about evolving landscape of data integrity, where GenAI can compromise data quality, mislead researchers, and influence downstream findings in fields such as health, politics, and social behavior. By surfacing detection strategies and empirical evidence of GenAI's impact, we aim to contribute to ongoing conversation about safeguarding research integrity and supporting scholars navigating these methodological and ethical challenges.