RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image Generation

Abstract: Subject-driven text-to-image (T2I) generation aims to produce images that align with a given textual description, while preserving the visual identity from a referenced subject image. Despite its broad downstream applicability - ranging from enhanced personalization in image generation to consistent character representation in video rendering - progress in this field is limited by the lack of reliable automatic evaluation. Existing methods either assess only one aspect of the task (i.e., textual alignment or subject preservation), misalign with human judgments, or rely on costly API-based evaluation. To address this gap, we introduce RefVNLI, a cost-effective metric that evaluates both textual alignment and subject preservation in a single run. Trained on a large-scale dataset derived from video-reasoning benchmarks and image perturbations, RefVNLI outperforms or statistically matches existing baselines across multiple benchmarks and subject categories (e.g., \emph{Animal}, \emph{Object}), achieving up to 6.4-point gains in textual alignment and 5.9-point gains in subject preservation.

• The paper introduces a novel metric that jointly measures textual alignment and subject consistency in text-to-image generation.
• It employs a dual classification approach using video-derived training data and refined prompt-image pairs to capture identity-preserving features.
• The research demonstrates improved performance over baselines with extensive benchmarks and ablation studies validating its scalable, cost-effective evaluation.

Evaluation of Subject-driven Text-to-Image Generation

The paper "RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image Generation" presents a novel metric designed to address the limitations in evaluating subject-driven text-to-image (T2I) generation models. This aim is to provide a reliable, cost-effective solution to measure both textual alignment and subject consistency while aligning closely with human judgments.

Introduction to Subject-driven T2I Generation

Subject-driven T2I generation models produce images based on a textual description while maintaining the visual identity from a reference image. The applications span personalized image creation, video character consistency, and verification in image retrieval tasks. The paper challenges the insufficiency of existing methods which either narrowly focus on textual or subject elements, are not human-aligned, or are cost-prohibitive when applied in large-scale evaluations. A new metric, proposed within the paper, evaluates these dual aspects from a single classification perspective, leveraging a large dataset derived from video-reasoning benchmarks and image perturbations. This innovation yields improvements in textual and subject-related benchmarks.

Methodology

Training Dataset Construction: The dataset creation involves the generation of training instances from video frames (Figure 1), where subject consistency classification utilizes positive and negative pairs extracted from video scenes showing identical entities. To foster model sensitivity to identity-defining attributes, image modifications via masking and inpainting are employed (Figure 2), forming part of the robust training dataset.

Model Training: Fine-tuning of PaliGemma, a 3B Vision-LLM, is central to the approach. This model processes inputs in the form of two sequential binary classifications for textual alignment and subject preservation.

Prompt and Image Generation: Prompt-image pairs are essential for textual alignment sensitivity. The prompts are auto-generated and include positive, negative, and hard negatives (Figure 3), ensuring comprehensive training coverage across potential model-use scenarios.

Figure 4: Illustration assessing both subject consistency and textual alignment through a tiered process incorporating identity-preserving and identity-altering assessments.

Performance Evaluation

The proposed tool is evaluated across multiple benchmarks, including DreamBench++, ImagenHub, and KITTEN. The results indicate up to 6.4-point gains in textual alignment and 8.5-point gains in subject consistency over existing baselines.

• Textual Alignment: The method achieved superior or competitive results compared to major baselines like CLIP, BLIPScore, and DreamBench++.
• Subject Consistency: On subject preservation tasks like those involving multi-subject settings, the metric consistently performed at a high level, surpassing API-heavy solutions like DreamBench++.

  Figure 5: Performance comparison in subject consistency (SC) and textual alignment (TA), showing superiority in distinguishing identity-agnostic and identity-defining changes.

Ablation Studies

Detailed ablation studies validate the design decisions underpinning the model. These studies examined the effectiveness of the classification order, subject markup utility, and image concatenation versus independent image processing. Results indicate maintaining textual alignment classification before subject consistency optimizes performance.

Figure 6: Example performance on ImageRAG rare entities, aligning closely with human preferences.

Conclusion

This work introduces a scalable and effective approach for evaluating T2I generation models, focusing on dual aspects crucial for subject-driven tasks. The integrated evaluation framework balances efficiency and fidelity, aligning with human judgment while avoiding costly alternatives. Future research aims to bridge current limitations in handling diverse styles and explicit identity modifications, guiding personalized image generation advancements.

In summary, the paper makes significant strides in filling the evaluation gap within T2I frameworks, promising enhanced accuracy and user satisfaction across applications in AI-driven image synthesis.