ArtQuant: Multimodal Artistic Aesthetics

• ArtQuant is a two-stage framework for artistic image aesthetics assessment that jointly models perceptual, cognitive, and emotional dimensions.
• It leverages a large-scale LLM-curated RAD dataset and integrates hierarchical description generation with a score prediction head to achieve state-of-the-art performance.
• The framework demonstrates robust empirical results in cross-domain artwork evaluation by minimizing prediction entropy through efficient multi-task aesthetic training.

ArtQuant is a two-stage multimodal framework for artistic image aesthetics assessment that unifies hierarchical description generation and continuous aesthetic score prediction. Built atop a large-scale, LLM-curated dataset (RAD), ArtQuant addresses both the data scarcity of multidimensional aesthetic annotations and the fragmentation present in conventional models by jointly modeling perceptual, cognitive, and emotional dimensions. Its mathematical grounding demonstrates provable reductions in prediction entropy, leading to robust convergence and empirical state-of-the-art performance in cross-domain artwork evaluation (Liu et al., 29 Dec 2025).

1. Architectural Overview

ArtQuant consists of four core components:

• Visual Encoder ($E_v$): A vision transformer (e.g., CLIP-ViT backbone) encodes an input image $x_v$ into a latent vector $z \in \mathbb{R}^d$.
• LLM Decoder ($G_{dec}$): A LLM (e.g., LLaMA-2) equipped with cross-attention mechanisms, using $z$ and auxiliary tokens as keys to generate hierarchical textual descriptions.
• Joint Description Generation Module: During supervised fine-tuning on RAD, this module prompts the LLM to generate multi-level aesthetic commentaries (perceptual, cognitive, emotional).
• Score Prediction Head: The LLM’s text-generation head, conditionally activated by special prefix tokens, outputs a soft distribution over discrete aesthetic “levels” $\{l_i\}$; the final score $x$ is computed as $x = \sum_i p_i l_i$, where $p_i$ represents the probability of level $i$.

Interaction proceeds as follows:

1. Input image $x_v \rightarrow E_v \rightarrow z$
2. $z$, dataset statistics $(\mu, \sigma)$, and aesthetic template tokens $\rightarrow$ Description Generator (cross-entropy optimization)
3. $z$ plus score-prefix tokens $\rightarrow$ Score Prediction Head $\rightarrow$ Level distribution $\{p_i\}$, compute score expectation
4. Multi-Task Aesthetic Training (MAT) stage: description and score losses optimized jointly in early epochs; only score loss in later epochs

2. Mathematical Formalism and Loss Design

ArtQuant’s loss functions integrate both description generation and score prediction:

• Description Generation Loss:

$$\mathcal{L}_{\text{CE}} = -\sum_{t=1}^T \log P(t_t \mid t_{<t}, z)$$

• Score Distribution KL Loss:

$$\mathcal{L}_{\text{KL}} = \sum_i p_i^{\text{gt}} \log\left( \frac{p_i^{\text{gt}}}{p_i^{\text{pred}}} \right)$$

where $$p_i^{\text{gt}} = \int_{l_i - d/2}^{l_i + d/2} \mathcal{N}(x; \mu, \sigma^2) dx$$.

• Aggregate Score Loss:

$$\mathcal{L}_{\text{ASL}} = \mathcal{L}_{\text{CE}}(\text{score-prefix}) + \lambda \mathcal{L}_{\text{KL}}$$

• MAT Stage Loss:

$$\mathcal{L}_{\text{MAT}} = \mathcal{L}_{\text{CE}}(\text{desc}) + \alpha \mathcal{L}_{\text{ASL}}$$

MAT is only used for early epochs; later epochs optimize $\mathcal{L}_{\text{ASL}}$ exclusively.

Theoretical Entropy Bounds

Let $Y$ be the discrete score levels, $D$ the generated description, and $Z$ the latent representation.

• Theorem 1 (Description–Score Dependency):

$H(Y \mid Z) \leq H(D \mid Z) + H(Y \mid D, Z)$

• Theorem 2 (Conditional Independence): If $Y \perp Z \mid D$,

$H(Y \mid Z) \leq H(D \mid Z) + H(Y \mid D)$

• Theorem 3 ($\epsilon$-Approximate Independence):

$$H(Y \mid Z) \leq H(Y \mid D) + \epsilon \log|Y| + H_2(\epsilon) + H(D \mid Z)$$

In ArtQuant, $H(Y \mid D)$ (“description sufficiency”) is bounded by RAD templates and LLM-based curation. $H(D \mid Z)$ (“generation ability”) is reduced via $\mathcal{L}_{\text{CE}}(\text{desc})$.

3. RAD Dataset Construction Pipeline

The Refined Aesthetic Description (RAD) dataset contains 70,000 triplets of artistic images, continuous aesthetic scores, and triple-level structured descriptions. Data is generated using a scalable, iterative LLM-based loop:

Input: image set I, human MOS scores {s}, templates T, dataset stats (μ,σ) Output: RAD = {(x_v, s, D)} for each x_v ∈ I: 1. s′ = normalize(s; μ,σ) 2. D_gen = GPT-4o.generate([“Score s′, stats μ,σ, template T”]) 3. align = DeepSeek-chat.judge(D_gen, s′) if align ≥ τ: add (x_v, s, D_gen) to RAD else: optionally regenerate up to N times end

Key aspects include conditioning LLM generation on both normalized score and dataset statistics to mitigate bias, multi-level templates prompting for perceptual, cognitive, and emotional analysis, and a discriminator (DeepSeek-chat) enforcing score–text consistency.

4. Empirical Results and Training Protocol

Experiments on standard artistic image assessment datasets—APDD (paintings), BAID (diverse artworks), and VAPS (historical art)—demonstrate ArtQuant’s empirical strengths. Evaluation metrics include SROCC and PLCC.

Dataset	ArtQuant (SROCC/PLCC)	Best Prior (Method)
APDD	0.871 / 0.894	0.810/0.840 (ArtCLIP)
BAID	0.543 / 0.589	0.533/0.583 (PVAFE)
VAPS	0.625 / 0.681	0.579/0.638 (AKA-Net)

Training is performed with a two-stage schedule, MAT (multi-task) followed by fine-tuning using score loss only:

• APDD: 3 epochs MAT + 1 fine-tune epoch
• BAID: 1 epoch MAT + 1 fine-tune epoch
• VAPS: 1 epoch MAT + 7 fine-tune epochs

Total training epochs are approximately 33% those of typical specialist models, leveraging strong LLM priors and MAT for efficient convergence. For BAID, 4 epochs require approximately 7.2 GPU-hours, outperforming prior approaches such as SAAN (48 hours on 3090).

5. Scalability, Cost, and Deployment Considerations

The RAD dataset’s construction pipeline achieves annotation scalability with negligible human labor, requiring up to three LLM calls and one discriminator pass per sample. Generation cost is near $0$ per annotation compared to manual annotation rates of \$1–\$5 per comment. ArtQuant’s faster convergence—one third the epochs of specialist models—is attributed to strong LLM priors and the MAT regime.

A public codebase and the RAD dataset are scheduled for release via the project’s GitHub page to facilitate future research.

6. Significance and Theoretical Context

ArtQuant systematically bridges the “cognitive gap” in artistic image aesthetics assessment by coupling hierarchical, multi-dimensional description with continuous score prediction in a unified, entropy-minimizing formalism. The approach is motivated by the inherent complexity of aesthetic evaluation, which intersects visual perception, cognition, and emotion. By operationalizing this complexity through joint training on a richly annotated, scalable dataset and grounding its design in theoretical entropy bounds, ArtQuant delivers empirically superior and computationally efficient solutions to artistic image quality assessment (Liu et al., 29 Dec 2025).

A plausible implication is that the ArtQuant paradigm may inform broader research into subjective, multidimensional evaluation tasks in vision-LLMs, especially where annotation cost and descriptive richness are bottlenecks.