Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluation

Abstract: LLMs are known to exhibit various forms of cultural bias in decision-making tasks, yet much less is known about their degree of cultural familiarity in open-ended text generation tasks. In this paper, we introduce the task of culturally-adapted art description generation, where models describe artworks for audiences from different cultural groups who vary in their familiarity with the cultural symbols and narratives embedded in the artwork. To evaluate cultural competence in this pragmatic generation task, we propose a framework based on culturally grounded question answering. We find that base models are only marginally adequate for this task, but, through a pragmatic speaker model, we can improve simulated listener comprehension by up to 8.2%. A human study further confirms that the model with higher pragmatic competence is rated as more helpful for comprehension by 8.0%.

• The paper demonstrates that integrating theory-of-mind modeling into artwork description generation can enhance cultural comprehension, with up to an 8.2% accuracy improvement in QA tests.
• It employs a visual-language model augmented with a self-evaluating mechanism that selects culturally-attuned descriptions based on simulated listener QA performance.
• The study highlights that adapting descriptions for cultural contexts significantly improves user understanding, paving the way for personalized applications in heritage and museum curation.

Culturally-Adaptive Pragmatic Generation in Artwork Description: A Technical Synthesis

Task Formulation and Motivation

This work introduces culturally-adapted artwork description generation, where the objective is to produce descriptions that facilitate comprehension for audiences from different cultural groups. The central motivation arises from the observation that visual artworks often contain implicit cultural symbols and narratives—elements that can be straightforward to interpret for in-group audiences, but are opaque to out-group listeners. Therefore, the challenge is not merely semantic or visual understanding but adapting generative outputs to the listener's cultural background, an explicit test of pragmatic competence.

A key illustration (Figure 1) showcases a Visual-LLM (VLM, specifically Gemma3) generating a description omitting the symbolic significance of the lotus leaf in a Chinese artwork, thus failing to assist culturally-unfamiliar users in answering tailored questions.

Figure 1: Example of a Gemma3-generated description of artwork for audiences unfamiliar with the cultural context. The description fails to explain the symbolism of the lotus leaf and therefore lacks evidence to help them answer the culturally-attuned question.

The task is concretely defined by parametrizing prompts to the VLM with basic metadata and an explicit cultural group indication. The output, a 75-word English description, must maximize interpretability for the target audience. The constructed dataset leverages over 6,000 artworks and synthesizes detailed backgrounds using GPT-5, with symbol-centric QA triplets per artwork supporting evaluation.

Evaluation Framework: Question Answering as Proxy for Pragmatic Competence

Evaluating open-ended, culturally-conditioned generation is a nontrivial problem—standard metrics (e.g., BLEU) are insufficiently sensitive to the pragmatics of interpretation. The proposed solution is to operationalize listener comprehension through a question-answering (QA) evaluation pipeline: for each generated description, a simulated listener (via an external VLM, e.g., Qwen2.5-VL) is prompted to answer culturally-attuned multiple-choice questions that probe the relevance and explanatory power of the description.

To disambiguate the role of descriptions versus prior cultural knowledge, the evaluation distinguishes between culturally-agnostic and culturally-attuned questions and partitions results by whether the listener is culturally familiar or unfamiliar with the artwork. Chain-of-thought prompting is employed to improve the reliability of simulated entailment reasoning, mitigating the tendency to ignore the provided description or cultural group defaults.

Theory-of-Mind Modeling for Self-Improving Pragmatic Generation

The principal methodological contribution is extending pragmatic modeling by incorporating theory-of-mind (ToM) reasoning into the generation process, inspired by the Rational Speech Act (RSA) framework. Instead of accepting the base description from persona-prompted generation, a pragmatic speaker samples multiple candidate descriptions and internally simulates a listener (using the same model) to assess which description maximizes the likelihood that the target audience would answer culturally-attuned questions correctly.

This process is illustrated in Figure 2. The top-ranked description, according to this self-evaluation, is then selected for presentation to human or external simulated listeners. An additional regularization (description likelihood) balances informativeness against hallucination or excessive verbosity.

Figure 2: Self-improving speaker mechanism—sampling descriptions, simulating listener QA, and selecting the candidate maximizing simulated comprehension.

Empirical Results: Quantitative and Subjective Analysis

Simulated Listener QA: Quantitative Gains

In simulated QA, pragmatic speaker models derived from Gemma3, LLaMA3.2-Vision, and LLaVA-OneVision outperform their corresponding base versions for unfamiliar audiences on culturally-attuned questions, with up to 8.2% improvement in accuracy. For instance, pragmatic Gemma elevates accuracy from 73.2% to 79.2%. Strong performance on agnostic questions is invariant to speaker model, confirming the specificity of the pragmatic improvements.

Human Subjectivity: Preferences and Knowledge Transfer

A human evaluation protocol engages participants (10 each from U.S. and Chinese backgrounds), exposing them to both base and pragmatic speaker descriptions as well as symbol-focused QA and subjective judgments. Participants consistently prefer pragmatic speaker outputs for comprehension and new symbol explanation, with a significant preference rate gap (8%).

Figure 3: Human preference rates show pragmatic descriptions improve user comprehension and introduce new information, but redundantly explain known content.

Figure 4: Example where pragmatic model enables unfamiliar audiences (and their simulated proxy) to answer cultural questions correctly, contrasting with the uninformative base output.

Crucially, the improvement is specific to knowledge gaps—pragmatic descriptions sometimes redundantly provide information already known to the user, indicating an over-approximation of knowledge mismatch.

Qualitative Divergence Between Models and Human Comprehension

Qualitative analyses expose divergences between simulated listener judgments and human understanding, highlighting systematic mismatches when descriptions lack actionable evidence or when external VLMs over- or under-infer based on cultural defaults (Figure 4). Correlation analysis reveals that models' self-evaluation scores more tightly track gains in human QA accuracy than subjective preference ratings, motivating further alignment work.

Human Study Flow and Evaluation Protocol

Figures 5–9 detail the multistage human evaluation design: initial symbol identification, pre-description QA, exposure to a (randomized) base/pragmatic description, post-description QA, and comparative preference judgments.

Figure 5: Human evaluation pipeline—participant instructions and task overview.

Figure 6: Participants select cultural symbols from a list, unaided by descriptions.

Figure 7: Participants answer detailed symbol questions, without any description support.

Figure 8: Participants receive a sampled model description and repeat the symbol tasks.

Figure 9: Side-by-side preference judgment—participants rate base versus pragmatic outputs.

Implications and Future Research

This study establishes that principled ToM-driven pragmatic modeling allows VLMs to incrementally improve in the adaptation of generated explanations. Critically, measurable gains in comprehension for out-group listeners are observed both in simulation and, to a slightly lesser degree, in controlled human studies. The research highlights challenges in aligning simulated evaluations with genuine human comprehension, especially given varied user priors and background knowledge. The work also exposes the limitation that pragmatic reasoning as deployed here can redundantly explain known concepts, indicating a need for dynamic estimation of user knowledge.

From a practical perspective, the framework provides a transferable pipeline for culturally-grounded evaluation of multimodal generation and establishes a template for deploying adaptable LLM and VLM systems in heritage, education, and museum curation applications across cultures. Theoretically, this work demonstrates the feasibility of recursive reasoning-based self-improvement and points toward more granular, perhaps user-interactive, modeling of prior knowledge.

Speculatively, next steps may include: explicit integration of user models tracking previously observed knowledge; few-shot personalization using feedback; as well as richer alignment methods that bridge between simulated and true human evaluation in real deployment, potentially leveraging interactive feedback loops or user persona induction (2604.02557).

Conclusion

This research introduces and formalizes culturally-adapted artwork description as a pragmatic generation challenge, proposing an evaluation protocol grounded in cultural-symbol-focused QA and establishing that VLMs can self-improve in pragmatic competence via theory-of-mind modeling. While both quantitative and subjective results validate the approach, the need for more precise modeling of listener priors and improved alignment between simulation and lived comprehension remains a central avenue for further work.