Academically intelligent LLMs are not necessarily socially intelligent

Abstract: The academic intelligence of LLMs has made remarkable progress in recent times, but their social intelligence performance remains unclear. Inspired by established human social intelligence frameworks, particularly Daniel Goleman's social intelligence theory, we have developed a standardized social intelligence test based on real-world social scenarios to comprehensively assess the social intelligence of LLMs, termed as the Situational Evaluation of Social Intelligence (SESI). We conducted an extensive evaluation with 13 recent popular and state-of-art LLM agents on SESI. The results indicate the social intelligence of LLMs still has significant room for improvement, with superficially friendliness as a primary reason for errors. Moreover, there exists a relatively low correlation between the social intelligence and academic intelligence exhibited by LLMs, suggesting that social intelligence is distinct from academic intelligence for LLMs. Additionally, while it is observed that LLMs can't ``understand'' what social intelligence is, their social intelligence, similar to that of humans, is influenced by social factors.

• The paper demonstrates that LLMs excel in academic tasks but struggle with nuanced social interactions as measured by SESI.
• It finds a weak correlation between academic and social intelligence, highlighting LLMs' inability to adapt to real social cues.
• The study shows that personality traits and assigned social roles can enhance LLMs’ social performance, suggesting new training paradigms.

Academically Intelligent LLMs Are Not Necessarily Socially Intelligent

Introduction

This paper investigates the social intelligence capabilities of LLMs despite their demonstrated prowess in academic intelligence tasks. Through the Situational Evaluation of Social Intelligence (SESI), the authors measure LLMs' ability to handle real-world social scenarios. The study reveals that LLMs display notable deficiencies in social intelligence and limited correlation between academic and social intelligence, advocating for the independent investigation of these two forms of intelligence.

Overview of Social Intelligence Evaluation

Inspired by human social intelligence frameworks, particularly Daniel Goleman's theory, the SESI benchmark was designed to evaluate LLMs' performance in authentic social situations derived from user interactions on platforms like Reddit. SESI tests five subcategories of social intelligence: empathy, social cognition, self-presentation, influence, and concern.

Figure 1: Overview of the situational evaluation of social intelligence.

The SESI benchmark addresses the limitations of existing evaluations by including comprehensive scenarios and dynamic real-world contexts, avoiding the pitfalls of static datasets and potential overfitting in LLMs' training.

Analysis of LLMs' Social Intelligence Performance

Thirteen state-of-the-art LLMs were assessed, revealing that social intelligence is a distinct construct from academic intelligence. The correlation coefficient between SESI scores and academic intelligence metrics was notably lower than that among academic benchmarks alone, affirming that social intelligence requires separate attention.

Figure 2: Heatmap for correlation matrix for social and academic intelligence measures.

LLMs demonstrate a tendency toward superficial friendliness, characterized by fixed pattern responses, which lack adaptability to varied social contexts (Figure 3). Additionally, LLMs show a misunderstanding of social intelligence prompts, often performing worse with higher supposed levels of social intelligence, indicating a fundamental misconception.

Figure 4: Change Ratio in the social intelligence performance of LLM agents following the manipulation of factors.

Influence of Social Factors on LLMs

The study examines the impacts of personality, gender, social role, and perspective on LLMs' social intelligence. LLMs with low agreeableness and high extraversion tend to perform better, likely because these traits mitigate the overly friendly bias of LLMs. Similarly, explicitly assigning a male gender or specific social roles like 'saler' improves LLMs' social intelligence. These effects suggest that aligning LLMs' characteristics with real-world stereotypes can enhance their social responses.

Figure 5: Impact of social factors on social intelligence (SI) performance of LLM agents.

SESI Benchmark Characteristics

SESI provides long, intricate contexts and diverse questions that require nuanced understanding and application of social intelligence (Figure 6). It presents a comprehensive and balanced assessment of social cognitive processes and social facility capabilities, setting it apart from other benchmarks which predominantly focus on social awareness alone.

Figure 6: SESI benchmark statistics.

Conclusion

The research underscores that social intelligence is not inherently aligned with academic intelligence in LLMs, suggesting the need for distinct training paradigms and benchmarks. As LLMs continue to integrate into socially interactive applications, understanding and enhancing their social capabilities remains crucial for reliable and effective human-AI interactions. Future work should focus on refining social intelligence benchmarks and exploring unique training methodologies to improve LLMs' understanding of complex social dynamics.