VaseVLM: Advanced 3D Cultural Analysis

• The paper introduces VaseVLM, a vision-language model that bridges data scarcity and cultural domain gaps in 3D analysis of ancient Greek vases.
• It leverages a novel 3D VQA dataset, reinforcement learning, and selective layer tuning to capture fine-grained geometric and semantic features.
• The model outperforms prior methods, showing significant improvements in retrieval accuracy and culturally informed descriptions as validated by expert evaluation.

VaseVLM is a vision-LLM (VLM) developed to address the challenges of analyzing ancient Greek pottery in 3D, with a focus on overcoming data scarcity and insufficient cultural domain expertise. Tailored specifically for the digital heritage domain, VaseVLM combines advanced multimodal architectures and domain-adaptive training, leveraging a novel 3D VQA dataset (VaseVQA-3D) and reinforcement learning frameworks to capture both the fine-grained geometric and semantic features that characterize archaeological artifacts. The model demonstrates significant improvements over previous state-of-the-art methods for ancient vase artifact analysis, enabling more accurate, culturally informed understanding and description of 3D heritage objects.

1. Motivations and Distinctives of VaseVLM

The principal objective of VaseVLM is to surmount two defining limitations in the application of VLMs to cultural heritage analysis: severe data scarcity and lack of specialized domain knowledge. Existing VLMs, though effective on general image captioning and visual question answering (VQA), fail to recognize the nuanced features—including shape, decoration, technique, and attribution—of rare, culturally significant objects such as ancient Greek vases. VaseVLM is specifically engineered to bridge this gap by:

• Leveraging a domain-specific, high-fidelity 3D dataset (VaseVQA-3D), paired with relevant question-answer data and captions enhanced by GPT-4o for archaeological accuracy.
• Utilizing a two-stage training strategy for improved extraction and reasoning over subtle archaeological cues.
• Advancing multimodal AI for digital heritage preservation through methodologies that span 3D reconstruction, semantic weighting, and cultural adaptation.

This approach distinguishes VaseVLM from baseline VLMs that rely on large-scale, generic data and lack sensitivity to expert-level archaeological descriptors (Zhang et al., 6 Oct 2025).

2. VaseVQA-3D Dataset and Data Pipeline

The VaseVQA-3D dataset is the first benchmark tailored for 3D visual question answering in the cultural heritage context. It comprises:

• 664 GLB-format 3D models of ancient Greek vases generated from an initial pool of over 30,000 2D images.
• 4,460 structured QA pairs, each addressing six core archaeological dimensions: Fabric, Technique, Shape, Dating, Decoration, and Attribution.
• Descriptive captions for every vase, constructed and refined for clarity and domain relevance via LLMs.

The data construction pipeline employs a three-stage filtering process:

Stage	Methodology	Output Quality Control
Quality Filtering	ResNet-50 classifier	Eliminates blurry/low-quality images
Semantic Filtering	CLIP-based retrieval	Discards vase fragments, selects optimal view
2D-to-3D Conversion	TripoSG (preferred)	Superior geometry and semantic fidelity compared to Hunyuan3D

This pipeline ensures that only museum-grade, semantically accurate 3D reconstructions are included, which is critical for reliable multimodal training and evaluation.

3. Model Architecture and Domain-Adaptive Training

VaseVLM builds upon the Qwen2.5-VL base model (3B and 7B variants), which is adapted via two targeted stages:

1. Supervised Fine-Tuning (SFT): LoRA is employed to fine-tune network weights while maintaining efficiency. Input sequences comprise 360-degree rotation videos of 3D vases and concise expert-informed captions.
2. Reinforcement Learning Optimization: The RLVR (Reinforcement Learning with Verifiable Rewards) framework applies the GRPO method. Outputs are decomposed into six semantic dimensions, each weighted for importance. Rewards are assigned using a cosine similarity measure; for dimension $i$:

$$r_i = \begin{cases} \text{sim}(g_i, t_i), & \text{if } \text{sim}(g_i, t_i) \geq \tau \ 0, & \text{otherwise} \end{cases}$$

where $g_i$ is the model’s generated output, $t_i$ the reference, and $\tau = 0.7$.

The full reward:

$R = \sum_{i=1}^6 w_i r_i - P + B$

with $w_i$ as weights, $P$ as penalties for poor outputs (length, repetition, irrelevance), and $B$ as a bonus for successful sequence matching. The entire reward is normalized to $[0, 1]$ for stability.

This methodology ensures that the model produces outputs that are not only semantically accurate but also archaeologically appropriate.

4. Performance Evaluation and Quantitative Benchmarks

Empirical results on the VaseVQA-3D dataset indicate:

• VaseVLM-7B-RL achieves a 12.8% improvement in top-1 retrieval accuracy (R@1) and a 6.6% boost in lexical similarity over previous state-of-the-art methods.
• Human evaluation by a panel of 10 archaeological experts corroborates the increase in descriptive and cultural accuracy for the reinforcement learning-optimized model variant.
• Comparisons to general-purpose VLMs and specialized 3D models (DiffuRank, Cap3D) reveal robust gains in FID, CLIP, and retrieval metrics, indicating superior recognition and understanding of 3D vase artifacts.

These improvements reflect the efficacy of domain-adaptive tuning and semantically weighted optimization for cultural heritage analytics.

5. Integration of Visual Region-Based Layer Tuning

VaseVLM can further benefit from optimal training and inference efficiency by adopting selective layer tuning strategies informed by the “visual region” paradigm (Wang et al., 2024):

• Selective updating of approximately 25% of layers, chosen by sparse and uniform distribution within the network, preserves nearly 99% of visual task performance and maintains linguistic capability.
• Training time and compute costs are reduced—e.g., a 23% time reduction demonstrated on LLaVA models.
• At inference, non-critical layers (outside the “visual region”) can be pruned using angular distance metrics, minimizing resource usage with little performance loss.

A plausible implication is that integrating visual region-based techniques, with careful selection of layers tailored to VaseVLM’s architecture, can yield additional computational efficiency without degrading the model’s archaeological reasoning or description quality.

6. Broader Implications and Remaining Challenges

VaseVLM introduces a technical advance for the intersection of computer vision, natural language processing, and digital archaeology:

• The methodologies enable more accurate documentation, restoration, and dissemination of complex heritage artifacts.
• Sets a reference standard for AI-driven cultural heritage analysis, making it feasible to extend similar workflows to other domains (e.g., sculpture, ancient coins).

Current limitations and future work include:

• Enhancing the data pipeline to further increase the retention rate and degree of fidelity in 3D reconstructions.
• Expanding the framework for broader cultural heritage applications beyond Greek pottery.
• Continuing to reduce training and inference demands without sacrificing quality.
• Exploring integration of additional modalities or domain knowledge for deeper archaeological interpretability.

This suggests a sustained research trajectory focused on refining multimodal AI for expertise-driven cultural artifact analytics, and on promoting computationally efficient yet semantically robust models for digital heritage preservation.