Donald Trumps in the Virtual Polls: Simulating and Predicting Public Opinions in Surveys Using Large Language Models

Abstract: In recent years, LLMs have attracted attention due to their ability to generate human-like text. As surveys and opinion polls remain key tools for gauging public attitudes, there is increasing interest in assessing whether LLMs can accurately replicate human responses. This study examines the potential of LLMs, specifically ChatGPT-4o, to replicate human responses in large-scale surveys and to predict election outcomes based on demographic data. Employing data from the World Values Survey (WVS) and the American National Election Studies (ANES), we assess the LLM's performance in two key tasks: simulating human responses and forecasting U.S. election results. In simulations, the LLM was tasked with generating synthetic responses for various socio-cultural and trust-related questions, demonstrating notable alignment with human response patterns across U.S.-China samples, though with some limitations on value-sensitive topics. In prediction tasks, the LLM was used to simulate voting behavior in past U.S. elections and predict the 2024 election outcome. Our findings show that the LLM replicates cultural differences effectively, exhibits in-sample predictive validity, and provides plausible out-of-sample forecasts, suggesting potential as a cost-effective supplement for survey-based research.

• The paper demonstrates LLMs' capability to simulate survey responses by effectively mirroring human value-based opinions in cross-cultural contexts.
• It employs ChatGPT-4o with demographic and historical data to forecast U.S. presidential election outcomes with improved accuracy using structural prompts.
• The study highlights a cost-effective research alternative while addressing LLM biases and methodological limitations in public opinion polling.

Analysis of LLMs in Survey Simulations and Election Predictions

The paper "Donald Trumps in the Virtual Polls: Simulating and Predicting Public Opinions in Surveys Using LLMs" thoroughly explores the potential of LLMs to replicate human responses in survey contexts and predict election outcomes. By leveraging ChatGPT-4o, the study provides an in-depth analysis of the model's capabilities across distinct yet interrelated tasks: simulating survey responses and forecasting electoral results.

Key Insights and Methodology

The research pivots on assessing ChatGPT-4o's effectiveness in capturing the nuances of human responses to value-based survey questions and its predictive accuracy regarding U.S. presidential elections. The study uses datasets from the World Values Survey (WVS) and the American National Election Studies (ANES), underscoring the model's engagement with two primary tasks:

1. Survey Simulation: The LLM generated synthetic responses to various socio-cultural and trust-related questions, reflecting responses from U.S. and China samples. The synthetic responses were then compared to actual human data to evaluate alignment in response patterns.
2. Election Prediction: ChatGPT-4o was employed to simulate voting behavior in past U.S. elections (2016 and 2020) and to project outcomes for the 2024 election based on demographic profiles extracted from the ANES data.

Evaluation and Findings

In the survey simulation task, the LLM demonstrated a remarkable ability to align with human response patterns, particularly when demographic inputs were factored in. However, the model occasionally overestimated the prevalence of socially progressive views in the U.S., although these biases were less pronounced in the Chinese dataset. This suggests a degree of cultural sensitivity, with the model accurately simulating cross-cultural differences in many instances. The ability of LLMs to maintain correlation structures in synthetic data comparable to human data reinforces their value in capturing complex social dynamics.

For election forecasting, the results showed varying success based on different prompts used. The structural prompt, which included additional state-specific electoral trends, afforded better alignment with actual past election outcomes compared to a simple role-play prompt. This illustrates LLMs' potential to offer insights into political climates and predict election results when calibrated with historical data adjustments.

Implications and Future Directions

The implications of integrating LLMs into survey-based research are significant. They present a cost-effective alternative for gathering and analyzing public opinion data, potentially alleviating the logistical challenges of traditional surveys. Moreover, their predictive capabilities can aid in exploring societal trends and electoral behaviors while allowing flexibility across cultural contexts.

Nevertheless, the study acknowledges limitations concerning LLM-specific biases and dataset diversity. Future research could extend these findings by incorporating more varied datasets and LLMs beyond ChatGPT-4o, exploring their applications across broader cultural and socio-political spectra. Addressing these areas will be crucial in refining methodologies and enhancing the LLMs’ efficacy as reliable tools in social science research and public opinion polling.

In conclusion, this paper illustrates the utility of LLMs in simulating and predicting human behavior, encouraging further development in this domain to optimize the use of artificial intelligence in understanding complex human dynamics.