PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents

Abstract: Psychological measurement is essential for mental health, self-understanding, and personal development. Traditional methods, such as self-report scales and psychologist interviews, often face challenges with engagement and accessibility. While game-based and LLM-based tools have been explored to improve user interest and automate assessment, they struggle to balance engagement with generalizability. In this work, we propose PsychoGAT (Psychological Game AgenTs) to achieve a generic gamification of psychological assessment. The main insight is that powerful LLMs can function both as adept psychologists and innovative game designers. By incorporating LLM agents into designated roles and carefully managing their interactions, PsychoGAT can transform any standardized scales into personalized and engaging interactive fiction games. To validate the proposed method, we conduct psychometric evaluations to assess its effectiveness and employ human evaluators to examine the generated content across various psychological constructs, including depression, cognitive distortions, and personality traits. Results demonstrate that PsychoGAT serves as an effective assessment tool, achieving statistically significant excellence in psychometric metrics such as reliability, convergent validity, and discriminant validity. Moreover, human evaluations confirm PsychoGAT's enhancements in content coherence, interactivity, interest, immersion, and satisfaction.

• The paper presents PsychoGAT, a framework that converts traditional self-report scales into engaging interactive fiction games using innovative LLM-based agents.
• It details a multi-agent methodology where designers, controllers, and critics collaboratively generate coherent and immersive narratives validated by key psychometric metrics.
• Experimental results demonstrate high reliability and enhanced user engagement for assessments of constructs like depression and personality traits.

PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents

Introduction

The paper introduces PsychoGAT, a paradigm that leverages LLMs to enhance psychological assessments through gamification. Traditional psychological assessment methods, such as self-report scales and psychological interviews, often suffer from engagement and accessibility issues. PsychoGAT addresses these challenges by embedding assessment into interactive fiction games, thus providing a more engaging and personalized experience for participants.

Figure 1: We propose PsychoGAT, a novel psychological assessment paradigm in the form of an interactive game instead of a traditional self-report scale.

Methodology

The PsychoGAT framework consists of several LLM-based agents responsible for different aspects of the assessment process: the game designer, controller, and critic. These agents collaboratively transform standard psychological scales into interactive narratives. The designer initiates the process by creating the game's settings and storyline based on the traditional scale. The controller generates and advises interactive content, while the critic refines these interactions to ensure coherence and engaging user experience.

Figure 2: The multi-agent framework of PsychoGAT. The designer generates settings for the interactive fiction game based on a given self-report scale. The controller, critic, and a human participant (or human simulator) engage in a cyclical interaction to facilitate the assessment process.

Experimental Validation

The paper validates PsychoGAT through extensive experiments across various psychological constructs, including depression and personality traits. The method achieves high psychometric reliability and validity, evidenced by metrics such as Cronbach's Alpha and convergent validity correlations. Furthermore, human evaluations indicate significant enhancements in user-related metrics, including coherence, interactivity, and immersion.

Figure 3: Comparisons among various assessment methods through automatic and human evaluations. All five methods are qualified for psychometric effectiveness, and PsychoGAT brings all-around user experience improvement. PsyMtrc= Psychometric, CH= Coherence, IA= Interactivity, INT= Interest, IM= Immersion, and ST= Satisfaction.

Implications and Future Work

PsychoGAT represents a significant step forward in the field of psychological assessment by integrating the versatility and capability of LLMs into interactive fiction formats. This approach not only enhances engagement but also provides a robust framework for assessing psychological states through non-traditional means. Future research may explore expanding PsychoGAT's application scope, further enhancing its adaptive storytelling capabilities, and incorporating broader psychological insights into the framework.

Figure 4: Word cloud representing extraversion and depression assessment games generated by PsychoGAT. The extraversion assessment focuses on social activities, whereas the depression assessment predominantly features thoughts and emotions.

Conclusion

PsychoGAT innovatively bridges the gap between traditional psychological assessments and interactive storytelling, leveraging LLM agents to offer immersive and effective diagnostic tools. Its demonstrated success in psychometric validation suggests that game-based assessments could be more widely adopted, providing alternatives to conventional methods and potentially reducing barriers to psychological evaluation accessibility.

Figure 5: Comparison of assessment paradigms. Traditional scales and psychologist role-playing interviews emphasize the recall of life events and self-reported feelings or thoughts. In contrast, PsychoGAT introduces an interactive fiction game environment where participants make decisions as the protagonist and craft their personal story.