Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

Abstract: Recent advances in vision-LLMs (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.

• The paper introduces Appear2Meaning, a benchmark that evaluates VLMs on predicting structured cultural metadata from heritage object images.
• It employs a rigorous two-stage curation and LLM-as-a-judge evaluation to assess metadata fields such as culture, period, origin, and creator.
• Results reveal low exact-match but higher partial-match rates, underscoring challenges in inferring non-observable, context-dependent cultural attributes.

Appear2Meaning: Structured Cultural Metadata Inference from Images

Motivation and Benchmark Design

The Appear2Meaning benchmark addresses a critical gap in multimodal AI by evaluating vision-LLMs (VLMs) on structured cultural metadata inference from heritage object images, as opposed to conventional perceptual captioning. Existing art and heritage caption datasets predominantly prioritize visual or emotive descriptions, lack cross-cultural object diversity, and do not systematically test models’ ability to infer non-observable attributes like creator, origin, and period. Appear2Meaning formalizes the task as latent attribute prediction from image-only inputs, requiring models to recover cultural, temporal, geographic, and creator metadata without access to textual prompts, ontology labels, or retrieval augmentation.

To support generalizable evaluation, the benchmark comprises 750 objects sampled from the Getty and Metropolitan Museum of Art, covering four major categories (ceramics, paintings, metalwork, sculpture) and four cultural regions (East Asia, Ancient Mediterranean, Europe, Americas). Data curation combines rule-based filtering with two-stage human verification for category and cultural assignments to mitigate provenance ambiguity and ensure balanced coverage.

Figure 1: Data curation pipeline combining rule-based filtering and human verification to build a balanced cross-cultural dataset.

Task Formulation and Evaluation Protocol

Appear2Meaning frames the task as a structured prediction problem: given an image $x$, the goal is to predict the metadata tuple $$\mathcal{M} = \{\text{culture},\text{period},\text{origin},\text{creator}\}$$. Each field corresponds to museum-controlled vocabularies or well-defined labels. The challenge lies in inferring non-observable attributes, as such metadata often depend on historical context, material provenance, and curatorial interpretation rather than purely visual cues.

The evaluation utilizes an LLM-as-a-Judge framework, where GPT-4.1-mini performs semantic alignment between model outputs and reference metadata at the attribute level, assigning correctness labels (correct, partial, incorrect). This enables fine-grained assessment of model capabilities by reporting exact-match accuracy, partial-match rates, and attribute-specific performance across cultures.

Vision-LLM Performance

Nine SOTA VLMs were evaluated, spanning both open-weight and closed-source architectures, including Qwen-VL-Max and Qwen3-VL (open), Pixtral-12B, GPT-4.1-mini, GPT-5.4-mini, and Claude Haiku 4.5. All models exhibited uniformly low exact-match accuracy ($\sim$0.01–0.03), indicating that coherent multi-attribute inference remains highly challenging. However, partial-match rates were substantially higher (up to 0.658), revealing that models frequently recover some metadata fields but rarely produce consistent, fully grounded profiles.

Attribute-level analysis shows that title and creator predictions outperform culture, period, and origin, reflecting the relatively stronger alignment of these fields with visual cues or frequent training patterns. Open-weight Qwen3-VL-Flash achieved dominant partial-match rates in East Asian and Mediterranean material (0.740, 0.706), and closed-source models (GPT-4.1-mini, Claude Haiku 4.5) showed no consistent advantage in cross-cultural inference.

Systematic Error Patterns and Case Studies

Model errors are typified by four dominant patterns:

• Cross-cultural misattribution: Models often anchor attribution to Eurocentric or well-represented training data priors, shifting American and Asian objects toward European origin. For example, the "Butter Pat" is consistently misclassified as French or Japanese porcelain across models—a phenomenon grounded in stylistic overlap and institutional biases.

  Figure 2: Example images for Object ID 1055_Butter Pat.

• Style transfer confusion: Visual resemblance, especially in ceramic techniques, drives models to assign American objects to Dutch, English, or European contexts, disregarding ground-truth provenance.

  Figure 3: Example images for Object ID 1513_Celery vase.

• Partial object recognition: Models reliably identify functional categories (e.g., fireplace tool) but systematically fail at cultural or region-level attribution, underscoring the gap between perceptual and contextual understanding.

  Figure 4: Example images for Object ID 42_Andiron.

• Over-specification and semantic drift: Models occasionally over-elaborate, introducing unsupported period or iconographic details, or provide plausible yet non-canonical titles and creators.

  Figure 5: Example images for Object ID 333_Basin.

Semantic alignment is further complicated by multi-view images and contextual complexity, where models generate plausible but non-exact interpretations of scene content and attribution.

Figure 6: Example images for Object ID 2b6e224c-686a-4b43-aa5a-1ef5520ef0ef. Top: full painting. Bottom: object back (left) and framed view (right).

Factors Contributing to Model Performance and Failure

Case study analysis demonstrates that observed deficiencies arise from both model and dataset factors:

• Training priors and stylistic biases: Models rely heavily on visual similarity and dominant associations from training data; museum collections with Eurocentric and Mediterranean predominance amplify attribution errors.
• Object-type and category imbalance: Recognition difficulty varies by region and category; sculpture and highly canonical forms present stronger cues for creator or period inference.
• Data quality and presentation: Variation in image resolution, object views, and metadata granularity directly impacts predictability; curatorial narrative may be essential but is often absent in structured evaluation.
• Epistemic limitations: Non-observable attributes frequently depend on historical and provenance records unavailable in image-only settings.

Implications and Future Directions

The benchmark demonstrates that current VLMs are inadequate for structured inference of cultural metadata from visual input alone. Partial success in creator and title fields highlights the importance of well-represented makers and functional categories, yet culture, period, and origin remain fundamentally elusive due to their reliance on contextual knowledge and historical documentation. These limitations have direct implications for real-world museum applications, provenance search, and automated heritage cataloging.

Future research will require integrating retrieval-augmented, ontology-grounded, and knowledge-based reasoning to bridge the gap between perceptual cues and structured cultural understanding, as well as expanding benchmarks to cover finer-grained, underrepresented, and hybrid cultural contexts. Multimodal learning must move beyond surface-level recognition to robust, multi-attribute reasoning, incorporating collection-level bias mitigation and human-in-the-loop evaluation.

Conclusion

Appear2Meaning offers a diagnostic benchmark probing the limits of VLMs in structured, cross-cultural metadata inference. Strong numerical results for partial-match rates juxtaposed with uniformly weak exact-match accuracy underscore the incapacity of perception-driven models to recover non-observable cultural attributes. The challenge is inherently epistemic and contextual; progress will depend on coupling visual intelligence with external knowledge integration and culturally sensitive evaluation protocols (2604.07338).