Assessing the Capabilities of LLMs in Humor:A Multi-dimensional Analysis of Oogiri Generation and Evaluation

Abstract: Computational humor is a frontier for creating advanced and engaging NLP applications, such as sophisticated dialogue systems. While previous studies have benchmarked the humor capabilities of LLMs, they have often relied on single-dimensional evaluations, such as judging whether something is simply ``funny.'' This paper argues that a multifaceted understanding of humor is necessary and addresses this gap by systematically evaluating LLMs through the lens of Oogiri, a form of Japanese improvisational comedy games. To achieve this, we expanded upon existing Oogiri datasets with data from new sources and then augmented the collection with Oogiri responses generated by LLMs. We then manually annotated this expanded collection with 5-point absolute ratings across six dimensions: Novelty, Clarity, Relevance, Intelligence, Empathy, and Overall Funniness. Using this dataset, we assessed the capabilities of state-of-the-art LLMs on two core tasks: their ability to generate creative Oogiri responses and their ability to evaluate the funniness of responses using a six-dimensional evaluation. Our results show that while LLMs can generate responses at a level between low- and mid-tier human performance, they exhibit a notable lack of Empathy. This deficit in Empathy helps explain their failure to replicate human humor assessment. Correlation analyses of human and model evaluation data further reveal a fundamental divergence in evaluation criteria: LLMs prioritize Novelty, whereas humans prioritize Empathy. We release our annotated corpus to the community to pave the way for the development of more emotionally intelligent and sophisticated conversational agents.

• The paper underscores that LLMs achieve mid-tier performance in Oogiri humor generation, with Gemini 2.5 Pro ranking highest among the models tested.
• The methodology employs a multi-dimensional rating system—covering Novelty, Clarity, Relevance, Intelligence, Empathy, and Overall Funniness—to benchmark humor outputs against human judgments.
• The results reveal a significant empathy gap and model biases such as positivity and self-preference, highlighting current limitations in replicating authentic human humor.

Multi-Dimensional Analysis of LLM Humor: Oogiri Generation and Evaluation

Introduction

This paper presents a systematic, multi-dimensional evaluation of the humor capabilities of contemporary LLMs, specifically targeting the Japanese improvisational comedy domain known as "Oogiri." Unlike prior computational humor studies that focus predominantly on single-dimensional funniness or binary classification, this work probes the rich structure of humor by annotating responses with six complementary dimensions: Novelty, Clarity, Relevance, Intelligence, Empathy, and Overall Funniness. The research investigates two core aspects: the extent to which LLMs can generate Oogiri responses judged as humorous by humans, and the degree to which LLMs' evaluative criteria align with human sensibilities.

Data Collection and Annotation Framework

A large-scale corpus was constructed by aggregating data from two established Oogiri platforms—Bokete (Oogiri-GO dataset) and Oogiri-Chaya. Each platform facilitated competitive punchline submissions to given topics, allowing for varied voting and evaluation dynamics. To mitigate systematic biases (e.g., bandwagon effects prominent in Bokete), strict filtering procedures were employed. Human-written responses spanning a wide spectrum of funniness were curated, alongside newly generated punchlines from three advanced LLMs: GPT-4.1, Gemini 2.5 Pro, and Claude Sonnet 4.

All responses were rated by native speakers along the six specified dimensions using a 0–4 Likert scale. Model-generated and serious (non-humorous) responses were included for robust baseline comparison. The annotation protocol ensured that each response type was assessed distinctly to control for anchoring effects.

Figure 1: Overview of the Oogiri experimental workflow, including prompt generation, LLM response synthesis, and subsequent multi-dimensional human and model evaluation.

Generation Performance Analysis (RQ1)

By benchmarking generated responses against stratified tiers of human performance (high, mid, low), the study finds that current LLMs achieve scores between those of human low and mid performers. Among tested models, Gemini 2.5 Pro ranks consistently highest across all dimensions, though the upper bound remains below human mid-tier responses in Overall Funniness.

A salient finding is the pronounced Empathy deficit in LLM outputs compared to humans, despite commensurate performance in Relevance, Clarity, and particularly Novelty. Empathy displays the largest performance gap, mirroring established cognitive and affective theories of humor which emphasize the importance of shared social context and emotional resonance. Human raters strongly associate Empathy with overall funniness, whereas LLMs gravitate towards Novelty as principal humor determinant.

Figure 2: Radar chart visualizing six-dimensional human evaluations of Oogiri responses across human tiers and LLMs. Empathy emerges as the weakest dimension for all LLMs, especially compared to high-tier human responses.

Detailed statistical analyses confirm these patterns. For instance, Gemini 2.5 Pro achieves a mean funniness score of 1.803, below the mid-tier average of 1.913, but well above the low-tier mean of 1.232. The gap in Empathy dimension between Gemini and top-rated humans exceeds 0.5 points, exceeding those for Clarity or Novelty.

Humor Evaluation: Alignment and Biases (RQ2)

Alignment quantification between LLM and human judgments utilizes Spearman and Pearson correlation metrics. For Overall Funniness, correlations remain low across all models ($\rho \approx 0.17$–$0.27$), with Claude Sonnet 4 marginally outperforming its peers but still exhibiting substantial divergence from human evaluative criteria.

Scatter plot visualizations and tierwise analyses reveal several persistent biases:

• Positivity Bias: LLMs frequently overrate unrelated or low-quality responses, granting funniness scores to content that human annotators rate near zero. Models demonstrate reluctance to issue negative ratings due to reinforcement learning procedures and design toward positive, cooperative feedback.
• Self-Preference Bias: LLMs systematically evaluate responses generated by other models at levels comparable to high-tier human answers, a phenomenon corroborated in recent evaluations of LLM-as-a-Judge paradigms.
• Dimension Divergence: While Empathy is the primary correlate of funniness for human judges (see heatmap analyses), LLM evaluations show strongest associations with Novelty. This confirms that LLM humor sensibility privileges unexpectedness and originality over shared subjective context.

  Figure 3: Human vs. GPT-4.1 ratings on Overall Funniness, with linear regression fits per response category. The dashed line indicates perfect agreement; LLM and human judgments show systemic disalignment, most notably for unrelated and low-tier responses.

  

  Figure 4: Spearman correlation matrix visualizing inter-dimensional correlations for human evaluations (lower triangle) and GPT-4.1 (upper triangle); Empathy shows the highest correlation with Overall Funniness for humans, while Novelty dominates for LLMs.

Notably, the only domain where LLMs approach robust alignment is in the classification of non-humorous, serious responses, suggesting that humor detection remains easier than subjective funniness ranking.

Influence of Topic Modality (Text vs. Image)

Analysis of prompt modality (text vs. image-based topics) exposes further nuances. Human evaluations consistently yield higher funniness for image-based topics, positing visual context as an amplifier for creativity. LLMs also benefit, but gains are model-specific and less pronounced. Some models, like Claude Sonnet 4, display an opposite trend, favoring text-based humor. These findings suggest that modality conditioning impacts not only generative capacity but also internal evaluative "preferences" of models, underscoring the need for tailored adaptation in humor-agent applications.

Implications and Future Directions

The empirical results delineate strict boundaries for contemporary LLM humor ability in the context of Japanese improvisational comedy. The inability of LLMs to adequately model Empathy, and their tendency to overrate novelty and penalize insufficient empathy, suggests that even advanced instruction and preference tuning are insufficient for bridging the human-AI humor gap. Model-specific biases (positivity, self-preference) must be actively mitigated to improve evaluation reliability.

From a theoretical standpoint, the multi-dimensional annotation methodology provides a new standard for computational humor research beyond binary or single-axis scoring. Practically, the released annotated corpus enables future integration of empathy-driven training objectives, potentially employing social commonsense reasoning modules, culture-specific tuning, or hierarchical reinforcement protocols aimed explicitly at emotion recognition and simulation.

Figure 5: Distribution histograms of human-rated funniness and companion LLM model evaluations, visualizing systematic rating differences and skewed model positivity bias.

Figure 6: Illustration of Oogiri prompting for LLM humor generation and evaluation, highlighting intercultural and instruction-induced context injection strategies.

Conclusion

By conducting granular, multi-dimensional benchmarking of LLMs on the complex Oogiri task, this study demonstrates that state-of-the-art models are capable of generating humor at a level between low and mid-tier humans, but fall short primarily due to insufficient empathetic reasoning. LLM humor evaluation is similarly limited, with empirical evidence of systemic biases, low agreement with human funniness criteria, and a marked prioritization of novelty over empathy. Future AI conversational agents that aspire to high-fidelity humor will require explicit empathy modeling, modality adaptation, and bias-aware evaluation design. The annotated corpus and methodology from this study pave the way for advanced development of emotionally intelligent humor systems.