PsychoGAT: LLM-Powered Game Assessment

• PsychoGAT is a novel paradigm for psychological measurement that uses LLMs to transform standard self-report scales into engaging, game-based assessments.
• It employs a three-agent framework—designer, controller, and critic—to generate interactive narratives while ensuring robust psychometric validity.
• Validation studies demonstrated high reliability and user engagement, outperforming traditional assessments in coherence and immersion.

PsychoGAT is a novel paradigm for psychological measurement that leverages LLMs to transform standardized self-report scales into engaging, interactive fiction games. Developed to address the challenges of low engagement, limited accessibility, and narrow generalizability in traditional assessment methods, PsychoGAT employs LLM agents in multiple functional roles to automate the generation, delivery, and refinement of game-based assessments that are psychometrically robust, fully automated, and generalizable across a wide range of psychological constructs (Yang et al., 2024).

1. Conceptual Framework and Motivation

Traditional psychological assessments predominantly rely on self-report scales—such as PHQ-9 for depression or MBTI subscales for personality—that are typically presented as static lists of items. These linear formats are frequently perceived as monotonous or repetitive, leading to common phenomena like inattentive responding, “straight-lining,” or noncompletion, particularly among nonclinical or younger populations. Moreover, clinician-led diagnostic interviews demand substantial expertise and resources, reflecting a global shortage of mental-health professionals. Electronic game-based or rule-based tools have made partial progress but are usually tailored for a single construct, requiring significant effort for adaption or expansion (Yang et al., 2024).

PsychoGAT introduces the core insight that LLMs can simultaneously serve as both psychologists and game designers. Specifically, LLM agents are organized into three roles:

• Game @@@@1@@@@: Interprets standard self-report scales and interweaves their items into a first-person interactive-fiction narrative.
• Game Controller Agent: Instantiates each redesigned scale item as a narrative node with branching decisions, each mapped to an underlying psychological trait indicator.
• Critic Agent: Iteratively refines the game controller’s output to maximize coherence, balance, and immersion.

Through this agent-based decomposition, PsychoGAT provides engagement comparable to immersive interactive fiction, accessibility by automating the assessment pipeline, and generalizability for arbitrary constructs via flexible, automated game generation.

2. System Architecture and Agent Functionalities

The PsychoGAT pipeline begins with user selection of a psychological construct and its corresponding validated scale. The process consists of the following key agent interactions:

1. Game Designer Agent
   • Inputs: Scale items $x_1, \ldots, x_k$, specified game type, and narrative topic.
   • Outputs:
     • Game title $T$.
     • Narrative outline $O = (O_1, \ldots, O_k)$.
     • Redesigned scale items $N = (N_1, \ldots, N_k)$. Each $N_i$ encodes a scenario with two response options, each mapped to a binary score.
2. Game Controller Agent ($\pi_c$)
   • For each node $N_i$, generates:
     • Paragraph $P_i$: The game text for the scenario.
     • Memory $M_i$: Summarized state up to turn $i$.
     • Two instructions $I_i^{\ell}, I_i^{r}$: Branching choices, each linked to one response option.
3. Critic Agent ($\pi_{\text{cri}}$)
   • Refines the controller output to $(P_i', M_i', I_i^{\ell\prime}, I_i^{r\prime})$ for up to three iterations or until approval for coherence, immersion, and neutrality.

Subsequently, a human participant or simulator makes sequential decisions, each mapped to their underlying score on the psychological trait. A hard-coded evaluator sums the binary scores to produce a total assessment score.

Agent	Function	Outputs
Designer	Translates scale, outlines narrative	$T$, $O$, $N_{1...k}$
Controller	Instantiates scenario, memory, instructions per node	$P_i$, $M_i$, $I_i^\ell$, $I_i^r$
Critic	Improves coherence, immersiveness, choice neutrality	$P_i'$, $M_i'$, $I_i^{\ell\prime}$, $I_i^{r\prime}$

The entire process is formally represented by iterative pseudocode for assessment execution, culminating in a final narrative log and summed score $S = \sum_{i=1}^k x_i$.

3. Transformation of Standardized Psychological Scales

PsychoGAT systematically converts standardized scale items into interactive narrative nodes:

• The game designer interprets each original scale item $x_i$ and constructs a narrative scenario $N_i$ embedding the construct in first-person fiction.
• For each $N_i$, two branching instructions are generated, each mapped to a binary trait indicator.
• The controller presents these instructions to the player. The player’s choice is directly mapped back to the scale’s original binary format.

Mathematically, for any $x_i \in \{0, 1\}$, selection of instruction $I_i^j$ yields $x_i = j$. The process produces a total scale score $S$. Although a probabilistic branching scheme with scene states $s_i$ and branching probabilities $p_{i,0}, p_{i,1}$ is conceivable, the present version employs strictly deterministic branching.

4. Psychometric Evaluation

PsychoGAT’s effectiveness was assessed through automated GPT-4-based participant simulation, summarizing its psychometric validity and reliability according to established metrics:

• Reliability:
  • Cronbach's $\alpha$: $$\alpha = \frac{k}{k-1}(1 - \frac{\sum_i \text{Var}(y_i)}{\text{Var}(X_{total})})$$
  • Guttman’s $\lambda_6$: $$\lambda_6 = 1 - \frac{\sum_i e_i^2}{\text{Var}(X_{total})}$$
• Construct Validity:
  • Convergent Validity: Pearson $r$ between PsychoGAT and its source scale ($|r| \geq 0.60$).
  • Discriminant Validity: Correlation with an unrelated construct (visual learning preference) is low ($|r| < 0.60$).

Twenty simulated runs per task (extroversion, depression, three cognitive distortions) provided the following results:

Task	Cronbach’s $\alpha$	Guttman’s $\lambda_6$	Convergent $r$	Discriminant $r$
Extroversion (MBTI-E)	0.97 (+++)	0.98 (+++)	0.97 (+)	–0.59 (+)
Depression (PHQ-9)	0.77 (+)	0.84 (++)	0.85 (+)	–0.07 (+)
All-or-nothing distortion	0.92 (+++)	0.93 (+++)	0.97 (+)	–0.44 (+)
Mind-reading distortion	0.92 (+++)	0.95 (+++)	0.97 (+)	0.25 (+)
Should-statements distortion	0.88 (++)	0.91 (+++)	0.93 (+)	–0.18 (+)

All tasks met professional psychometric standards ($\alpha \geq 0.70$, $|r_{\text{convergent}}| \geq 0.60$, $|r_{\text{discriminant}}| < 0.60$).

5. Human Evaluation and Comparative Analysis

Human rater studies assessed the narrative and engagement properties of PsychoGAT-generated games across multiple dimensions: coherence, interactivity, interest, immersion, and satisfaction.

• Participants: 33 English-proficient raters (aged 18–45, no current mental-health issues).
• Protocol: 15 “all-or-nothing” stories rated on 1–5 Likert scales, normalized to $[0.1, 0.9]$.

Results (normalized mean scores):

Method	Coherence	Interactivity	Interest	Immersion	Satisfaction
Traditional Scale	0.35	0.30	0.28	0.25	0.31
Auto-Scale	0.50	0.45	0.48	0.43	0.44
Psycho-Interview	0.55	0.50	0.52	0.50	0.49
DoT-Interview	0.58	0.54	0.56	0.55	0.53
PsychoGAT	0.80	0.77	0.79	0.78	0.76

Rater feedback characterized PsychoGAT as “immersive” and praised the neutrality and engagement of the branching instructions.

6. Strengths, Limitations, and Future Directions

PsychoGAT offers high engagement, full automation, and generality across psychological constructs, with psychometric properties rivalling or surpassing those of traditional assessments. Notable limitations include:

• LLM Hallucination: There is risk that the generated narratives could diverge from intended scale content without careful prompt engineering and agent architecture.
• Simulated Participant Bias: Participant simulation via GPT-4, while solving key ethical concerns, may not fully replicate actual user response patterns.
• Language and Cultural Scope: The current validation is restricted to English and simulated respondents, limiting generalizability to diverse populations.

Proposed avenues for further research include:

• Localization and cross-cultural validation through translation and psychometric re-evaluation.
• Clinical trials with real patients to assess longitudinal sensitivity to symptom change.
• Fine-tuning of LLMs with clinical interview transcripts to reduce hallucination rates.
• Exploration of hybrid text-graphics or video-game-based implementations to further increase immersion.

An illustrative example of PsychoGAT’s transformation capabilities can be found in the “Echoes of Auroria” adventure scenario, where MBTI-E items are rendered as sequential decision nodes in an evolving, first-person fantasy narrative, with trait assessment seamlessly embedded in gameplay (Yang et al., 2024).

In summary, PsychoGAT formalizes a flexible, LLM-powered framework for psychological measurement, uniting interactive fiction and rigorous psychometric evaluation in a scalable, engaging format.