Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting

Abstract: Automatic response forecasting for news media plays a crucial role in enabling content producers to efficiently predict the impact of news releases and prevent unexpected negative outcomes such as social conflict and moral injury. To effectively forecast responses, it is essential to develop measures that leverage the social dynamics and contextual information surrounding individuals, especially in cases where explicit profiles or historical actions of the users are limited (referred to as lurkers). As shown in a previous study, 97% of all tweets are produced by only the most active 25% of users. However, existing approaches have limited exploration of how to best process and utilize these important features. To address this gap, we propose a novel framework, named SocialSense, that leverages a LLM to induce a belief-centered graph on top of an existent social network, along with graph-based propagation to capture social dynamics. We hypothesize that the induced graph that bridges the gap between distant users who share similar beliefs allows the model to effectively capture the response patterns. Our method surpasses existing state-of-the-art in experimental evaluations for both zero-shot and supervised settings, demonstrating its effectiveness in response forecasting. Moreover, the analysis reveals the framework's capability to effectively handle unseen user and lurker scenarios, further highlighting its robustness and practical applicability.

• The paper presents SocialSense, a framework that leverages large language models for latent persona extraction to build belief-augmented social graphs for improved response forecasting.
• It integrates belief nodes and social interactions using graph neural networks, achieving up to 10.28% improvement in correlation metrics and 5.99% in F1 scores.
• The approach effectively forecasts responses for various user segments, including lurkers and unseen users, in both supervised and zero-shot settings.

Decoding the Silent Majority: Inducing Belief Augmented Social Graph with LLM for Response Forecasting

Introduction

The paper "Decoding the Silent Majority: Inducing Belief Augmented Social Graph with LLM for Response Forecasting" introduces SocialSense, a framework aimed at enhancing automatic response forecasting in news media scenarios. The primary objective is to predict responses to news messages by leveraging belief-augmented social networks. This approach addresses the limitation of existing methods that often overlook the subtle social dynamics and belief systems that influence how users interact with news content.

Methodology

SocialSense is built upon three key components:

1. Latent Persona Extraction: Utilizing LLMs, such as ChatGPT, latent personas of users are extracted by analyzing their social media profiles and historical posts. This includes identifying moral and human values, beliefs, and opinions that shape user behavior. The initial stage involves designing a specific prompt $\mathrm{P}_l$ to facilitate the extraction of this latent information.
2. Belief-Augmented Social Network: The framework develops a belief-centered network by integrating induced and existing social interaction data. The network structure comprises nodes representing news media, users, and beliefs, connected by edges indicating interactions. This structure emphasizes the importance of bridging users with similar beliefs, even if they reside in distant and disconnected social communities.
3. Graph Information Propagation: Utilizing Graph Neural Networks (GNN), information is propagated through the network to update user representations. A heterogeneous graph transformer mechanism captures interactions among varied node types, facilitating nuanced information flow and enhancing prediction accuracy.

   Figure 1: Distribution of belief values.

Task Formulation

The task is defined as a multi-class prediction problem. Given a user persona and a news message, the objective is to predict the user's sentiment polarity $\phi_p$ and intensity $\phi_{int}$. The framework aims to capture how users with similar beliefs respond to news, with evaluations conducted in both zero-shot and supervised settings.

Experimental Validation

SocialSense's effectiveness is validated through experiments on a curated dataset from Twitter, involving over 18,000 users. Evaluations were conducted to gauge the framework's performance in both supervised and zero-shot scenarios.

• Supervised Scenario: The framework showed superior performance over baseline models like DeBERTa and RoBERTa, achieving higher Spearman and Pearson correlations for sentiment intensity prediction, and better F1 scores for sentiment polarity predictions.
• Zero-Shot Scenario: By simulating graph propagation with social prompts, SocialSense$_\text{Zero}$ demonstrated enhanced prediction accuracy, underscoring the importance of incorporating social context in forecasts.

  Figure 2: Prompt template $\mathrm{P}_l$ used for extracting user latent profiles.

Results and Discussion

The results exhibit that integrating belief nodes within the social network significantly influences the model's performance. The proposed belief-centered approach outperforms competitors by a margin of 10.28% in correlation metrics and 5.99% in F1 scores, illuminating the importance of belief systems in user behavior forecasting.

• Lurker and Unseen User Scenarios: SocialSense excels in scenarios involving lurkers and unseen users, with improvements observed due to the belief-centered graph's ability to infer accurate user responses without extensive historical data.
• Ablation Studies: Component-level evaluations highlighted the vital roles of belief nodes, user-news edges, and the methodology of user representation initialization.

  Figure 3: Prompt template $\mathrm{P}_s$ used for aggregating neighbor information.

Conclusion

The proposed SocialSense framework provides a robust solution for automatic response forecasting. By bridging socially distant yet belief-aligned users, the framework captures intricate beliefs and social dynamics, thereby enhancing predictive accuracy. Future work may explore the generalization of this approach to different domains and the development of dynamic social prompting strategies for evolving social media data.

In summary, SocialSense represents a significant advancement in the understanding and prediction of user responses in the context of news media, driven by a detailed modeling of belief systems and social interactions.