Where do LLMs Fall Short in CBT-Guided Affective Reasoning?

Abstract: Cognitive Behavioral Therapy (CBT) provides a structured framework for understanding a user's mental state by examining the interaction between cognitive and behavioral factors. However, out-of-the-box LLMs respond fluently and empathetically, yet collapse into validation & reflection, regardless of what the user actually needs. They know theoretical CBT (scoring up to 96% accuracy on licensing exam questions) but fail to apply it effectively. We explore this gap with a knowledge-guided framework that treats CBT dialogue as controlled affective reasoning: user narratives are decomposed into Beck's Cognitive Conceptualization structure, grounded in clinical SNOMED CT concepts validated via Natural Language Inference, and a Multiple Chain-of-Thought (MCoT) strategy selection between Validation & Reflection, Socratic Questioning, or Alternative Perspectives. To measure whether such guidance actually changes behavior, we introduce the Protocol Leverage Force (F), a behavior-level metric that captures how far an intervention shifts a model away from its default response. Across three open-weight LLMs and 14 RealCBT-derived case studies, evaluated with human experts, valence-arousal trajectories, and linguistic entrainment, F shows that simply introducing protocol definitions via single chain-of-thought prompting fails to change LLM behavior, while MCoT on these definitions guides strategy selection better. Still, the effect stays within 1% (approx. 1.2-1.3%), and all models remain biased toward Validation & Reflection. These results show CBT knowledge alone does not ensure effective application, giving the affective-computing community instrumentation to measure where LLMs fall short.

• The paper introduces a framework that decomposes user narratives into clinical elements using SNOMED CT and contrastive NLI.
• The paper quantifies intervention shifts with the Protocol Leverage Force metric, revealing modest deviations (~1.2–1.3%) in LLM responses.
• The paper finds that despite high CBT exam accuracy, LLMs default to validation strategies, limiting effective applied therapeutic interventions.

CBT-Guided Affective Reasoning: Empirical Limits in LLMs

Background and Motivation

LLMs are increasingly utilized for mental health support amid high-stakes interactions at scale. While LLMs have demonstrated fluent, empathetic conversational abilities, they consistently default to validation or reflective responses regardless of the underlying client needs, demonstrating an "appraisal bias" and neglecting the structured reasoning required for effective therapeutic intervention. Although LLMs achieve up to 96% accuracy on theoretical CBT exam tasks, this proficiency does not translate to applied affective reasoning in real or simulated clinical settings.

Figure 1: Comparative response strategies of GPT-5.3 and an affective-reasoning LLM, highlighting the tendency of generic LLMs toward premature assumptions and problem-solving.

Framework: Clinical Structuring and Protocol-Guided Reasoning

The paper introduces a framework for controlled affective reasoning modeled after Beck's Cognitive Conceptualization Diagram (CCD), explicitly decomposing user narratives into triggers, automatic thoughts, emotions, and behaviors. This decomposition supports principled intervention selection by constraining model output to one of three CBT strategies: Validation (V), Socratic Questioning (SQ), or Alternative Perspective (AP).

Figure 2: The appraisal-based user cognitive model explicitly links triggers, thoughts, emotions, and behaviors for downstream affective reasoning.

Beyond cognitive modeling, user queries are grounded in clinical terminologies via SNOMED CT with semantic retrieval and contrastive NLI verification to ensure logical support for downstream reasoning. This systematic linkage of natural language to clinical entities mitigates surface-level interpretation and enforces semantic coherence.

Figure 3: Clinical grounding of user queries through semantic retrieval and NLI-based validation to categorize clinical concepts.

Multiple Chain-of-Thought and Strategy Selection

The proposed framework operationalizes a Multiple Chain-of-Thought (MCoT) workflow, generating explicit candidate responses for each CBT principle per turn and scoring them to select the intervention predicted to best advance the dialogue. This enables transparent analysis of intervention strategy adherence and efficacy, diverging from traditional single-chain prompting.

Figure 4: Parallel response generation for V, SQ, and AP, scored and selected per turn for protocol adherence.

Protocol Leverage Force: Novel Metric for Behavioral Reorientation

A new metric, Protocol Leverage Force ($\mathcal{F}$), quantifies the shift from a model’s baseline response when protocol-guided reasoning is introduced. $\mathcal{F}$ is sensitive to (i) the strength of strategy commitment, (ii) the degree of movement toward the patient's articulated cognitive state, and (iii) the embedding-space divergence from native responses, drawing conceptual inspiration from classical centrifugal force dynamics.

Figure 5: Schematic of $\mathcal{F}$ showing behavioral reorientation as a function of protocol-driven response deviation.

Across Gemma3-12B, Mistral-7B, and GPT-OSS-20B, $\mathcal{F}$ consistently revealed only modest shifts in response behavior (~1.2–1.3%), indicating that even with structured guidance, models remain strongly biased toward validation/reflection and fail to reliably apply Socratic Questioning or Alternative Perspective strategies.

Empirical Evaluation: Human, Affective, and Linguistic Metrics

Human expert review demonstrates consistently high recognition of validation strategies but ambiguous quality and poor inter-rater reliability. Socratic Questioning, though less frequent, yields greater evaluator consensus, suggesting higher interpretive reliability in downstream applications.

Entrainment metrics (nCLiD) and valence–arousal trajectories further reveal divergence from human therapist sessions. Synthetic sessions show increased linguistic alignment yet lack the affective depth and oscillatory nuance of RealCBT interactions.

Figure 6: Comparative valence–arousal trajectories of synthetic versus real CBT sessions, depicting affective arc differences and session length effects.

Protocol Adherence and Concept Usage

LLMs predominantly select Validation strategies, with statistically significant shift magnitudes only observed for non-default protocols. Human evaluators most often prefer Gemma3's protocol-guided responses, but agreement is weak, underscoring persistent subjectivity in assessing dialogic appropriateness.

Clinical concept grounding improves semantic alignment and interpretability, but models favor user cognitive model sources over SNOMED CT concepts, indicating insufficient utilization of structured clinical evidence.

Figure 7: Protocol adherence radar plots showing intervention strategy distribution across models, as judged by GPT-5.1.

Theoretical and Practical Implications

The findings demonstrate a persistent gap between theoretical CBT knowledge and its practical application by LLMs, independent of model architecture or prompting paradigm. This gap is quantifiable by $\mathcal{F}$ and reproducible across datasets and evaluation modes. The protocol-guided approach modestly improves strategy selection but does not fundamentally overcome appraisal bias or default empathetic responding.

From an affective computing perspective, the instrumentation developed herein provides a rigorous framework for diagnosing LLM shortcomings in clinical reasoning. Practical deployment requires deeper integration with real-world cognitive models and adaptive resource mapping (e.g., SBCCD), as well as expanded safety and consent mechanisms given the psychological privacy risks inherent to cognitive appraisal inference.

Future Directions

Closing the application gap necessitates (1) mechanistic tracing of model internal pathways for protocol-driven intervention, (2) multi-agent conversational paradigms simulating broader CBT trajectories, and (3) extension to more diverse and safety-critical case studies. Further research should explore hierarchical chaining across affective states and dynamic appraisal in long-form therapeutic dialogues. Integrating contextual grounding and resource stratification will be critical for clinically viable AI-assisted therapy.

Conclusion

The knowledge-guided CBT framework delivers measurable, interpretable improvements in protocol adherence and affective reasoning, but empirical results establish that CBT knowledge alone is insufficient for effective application in existing LLMs. The introduced behavioral metrics and structured guidance offer a pathway for systematic assessment and incremental improvement, yet the central challenge of moving models past their empathy bias remains unresolved. Achieving affect-aware systems capable of authentic CBT-guided dialogue will require advances at the intersection of mechanistic interpretability, clinical grounding, and adaptive reasoning.