Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations

Abstract: Understanding social interactions involving both verbal and non-verbal cues is essential for effectively interpreting social situations. However, most prior works on multimodal social cues focus predominantly on single-person behaviors or rely on holistic visual representations that are not aligned to utterances in multi-party environments. Consequently, they are limited in modeling the intricate dynamics of multi-party interactions. In this paper, we introduce three new challenging tasks to model the fine-grained dynamics between multiple people: speaking target identification, pronoun coreference resolution, and mentioned player prediction. We contribute extensive data annotations to curate these new challenges in social deduction game settings. Furthermore, we propose a novel multimodal baseline that leverages densely aligned language-visual representations by synchronizing visual features with their corresponding utterances. This facilitates concurrently capturing verbal and non-verbal cues pertinent to social reasoning. Experiments demonstrate the effectiveness of the proposed approach with densely aligned multimodal representations in modeling fine-grained social interactions. Project website: https://sangmin-git.github.io/projects/MMSI.

• The paper introduces innovative tasks like speaking target identification, pronoun resolution, and mentioned player prediction to capture multi-party social dynamics.
• It presents a baseline that synergizes language and visual cues by leveraging advanced pre-trained models for enriched interaction understanding.
• The study establishes robust benchmarks on social deduction games, paving the way for more socially aware AI applications in virtual assistance and robotics.

Advancing the Understanding of Multimodal Social Interactions

Introduction to the New Challenges in Social Interactions

In the domain of artificial intelligence and particularly within the study of multimodal social interaction, considerable progress has been achieved. Yet, most efforts have focused on analyzing the behavior of isolated individuals or have not sufficiently captured the dynamics of multi-party interactions through both verbal and non-verbal cues. Recognizing the gap in current methodologies, a recent study proposes to advance our understanding by introducing new tasks and a novel approach that emphasizes the need for densely aligned language-visual representations in modeling the dynamics of multi-party social interactions, particularly within the context of social deduction games.

Novel Tasks to Model Fine-Grained Social Dynamics

The paper introduces three innovative tasks aimed at enriching our perception of multimodal social interactions:

1. Speaking target identification,
2. Pronoun coreference resolution, and
3. Mentioned player prediction,

These tasks were meticulously designed to challenge the conventional boundaries of understanding in multi-party social interactions by focusing on the aligned interpretation of verbal utterances and non-verbal cues (e.g., gestures, gazes) within the domain of social deduction games.

Methodology and Approach

To address these tasks, the authors develop a multimodal baseline model that synergizes densely aligned language-visual representations. This model facilitates the concurrent analysis of spoken utterances and corresponding visual cues (including gestures and gaze directions) to provide a comprehensive understanding of social interactions. The model employs an innovative technique for aligning player visuals with their spoken references, enabling a nuanced analysis that had previously been unattainable with holistic visual representations or single modality analyses alone.

Through extensive data annotations within social deduction game datasets, this study not only creates robust challenges but also provides the necessary benchmarks to evaluate future approaches in this domain. The baseline model introduced leverages advanced machine learning techniques, including the use of pre-trained LLMs like BERT, RoBERTa, and ELECTRA, alongside innovative alignment strategies and visual feature extraction methods.

Key Findings and Contributions

The experimental results presented in the paper underscore the efficacy of the proposed approach, showcasing substantial improvements in task performance when leveraging the densely aligned multimodal representations over traditional or unimodal approaches. Specifically, these improvements are evidenced across all three introduced tasks, highlighting the significance of integrating both verbal and non-verbal cues through alignment in understanding social interactions.

Furthermore, the paper thoroughly examines the impact of various factors including visual feature types (gesture versus gaze features), the role of conversational context, and the effectiveness of permutation learning in enhancing model performance. These examinations shed light on the complex interplay between different elements of social interactions and underscore the nuanced understanding required to model them effectively.

Future Directions and Implications

This research not only sets a new benchmark for the study of multimodal social interactions but also lays down the foundation for future advancements in developing socially aware artificial intelligence. The introduction of densely aligned multimodal representations opens new avenues for exploring the intricacies of human social behavior, paving the way for more naturalistic and effective AI systems capable of navigating the social world. Moreover, by releasing the benchmarks and source code, the research facilitates further exploration and innovation in this rapidly evolving field.

The proposed approach and its success in enhancing our understanding of multimodal social interactions hold promise not only for the further development of social artificial intelligence but also for practical applications in areas such as virtual assistance, social robotics, and beyond, where an in-depth understanding of human social dynamics is crucial.

In conclusion, this paper makes a significant contribution to the burgeoning field of social artificial intelligence by introducing robust tasks, a novel methodology, and offering insightful findings that collectively advance our understanding of the complexities of multimodal social interactions.