MIBE: Multi-subject Interaction Benchmark and Evaluator for Personalized Image Generation

Abstract: Multi-subject personalized image generation requires the precise rendering of all requested reference identities and their specified interactions based on a guiding prompt. However, state-of-the-art models still struggle with this process, frequently omitting subjects, failing to preserve reference appearances, or misattributing interactions. Furthermore, existing metrics designed primarily for single-subject fidelity cannot reliably capture these errors, suffering severe degradation in ranking separability and failing to align with human preference as the subject count increases. To address this gap, we introduce Multi-subject Interaction Benchmark and Evaluator (MIBE), a unified framework comprising a Multi-subject Interaction Benchmark (MIB) and a Multi-subject Interaction Evaluator (MIE). MIB systematically covers diverse relation types and scene complexities through a decoupled data regime. This consists of a 60K-pair VLM-labeled Silver Set for scalable metric training and a 4K-pair double-blind Human Evaluation Gold Set covering a diverse range of state-of-the-art generators, with the Silver Set reaching 95.1% cross-VLM preference agreement. To demonstrate the utility of this benchmark, we present MIE, a lightweight, reference-conditioned evaluator trained exclusively on the Silver Set with a dual-head ranking and diagnosis objective. MIE exhibits strong cross-generator generalization on the Gold Set, achieving 0.922 overall pairwise accuracy against human preference, including 0.982 on seen generators and 0.884 on unseen generators. By outperforming a broad spectrum of baseline metrics, including CLIP and DINO variants, MIE demonstrates that diagnostic supervision can preserve ranking separability and human alignment where traditional evaluators collapse.

• The paper introduces MIBE as a unified framework combining a benchmark (MIB) and evaluator (MIE) to diagnose multi-subject binding failures.
• It employs hierarchical prompt design and dual-head evaluation to capture detailed errors in existence, appearance, and interaction.
• Empirical results reveal MIE outperforms existing metrics, achieving 92.2% pairwise accuracy and robust diagnostics across diverse models.

MIBE: A Rigorous Benchmark and Diagnostic Evaluator for Multi-Subject Personalized Image Generation

Introduction and Motivation

The key bottleneck in reference-conditioned image generation is the accurate assignment of multiple visually grounded identities and their interrelations—a problem the authors term the binding problem. Despite recent advances in personalized generation, current models degrade rapidly as subject count increases, frequently omitting identities, corrupting appearances, and misassigning role-conditioned interactions. Standard evaluation metrics such as CLIP and reward-model-based preferences, while aligned with aesthetic or coarse compositional features, systematically fail in crowded or heavily interactive setups. This shortfall is well-illustrated by the collapse of pairwise metric-human agreement to near-random as prompt subject count grows beyond two.

Figure 1: Pairwise agreement between automatic metrics and human judgment decays toward chance as the number of prompt-specified subjects increases, but MIE retains strong alignment even in dense scenarios.

Consequently, the field has lacked a robust, scalable framework for both benchmarking and diagnosing binding failures in complex personalized multi-subject scenes. The paper introduces MIBE as a solution—a unified architecture consisting of the Multi-subject Interaction Benchmark (MIB) and the Multi-subject Interaction Evaluator (MIE)—designed to fill this critical gap by supporting robust, interpretable, and human-aligned evaluation in high-complexity personalized image generation.

Benchmark Design: Factorized Complexity and Human-Grounded Annotation

MIB is constructed through strict hierarchical prompt-reduction and controlled reference pooling, ensuring that prompt complexity (entity count, human/non-human ratio, interaction class) is explicitly factorized. Each data instance specifies a prompt, a tuple of reference images for each subject, and a pair of candidate generations by distinct models. This decoupling enables precise attribution of failure sources.

Annotation in MIB comprises two regimes:

• Silver Set (60K pairs): SOP-guided dual-VLM consensus (Gemini 2.5/3.1) provides scalable preference and diagnostic labels, yielding high inter-judge agreement (95.1%). This enables efficient and robust training signal for learned evaluators.
• Gold Set (4K pairs): Double-blind human annotators deliver high-fidelity preference and structured diagnostic labels across a cross-section of strong contemporary generators, forming the primary testbed for model and metric evaluation.

The label protocol strictly distinguishes Existence (all subjects rendered distinctly), Appearance (subject-level identity/attribute preservation, no feature leakage), and Interaction (semantic and physical alignment of roles/actions/relations to prompt)—making fine-grained failure attribution feasible.

Evaluator Architecture: Dual-Head Human-Aligned Diagnostics

Core to the framework is MIE, a multimodal VLM-based evaluator conditioned on prompt, references, and generated candidate. Its dual-head design comprises:

• A scalar ranking head for pairwise preference prediction, driving leaderboard-style benchmarking.
• A diagnostic head jointly predicting Existence/Appearance/Interaction correctness per candidate, yielding interpretable, actionable feedback for model development and error analysis.

Joint optimization of ranking (margin loss) and diagnostic (binary cross-entropy) objectives (with weighting $\alpha$ and $\beta$) compels the ranking signal to be grounded in concrete binding errors, mitigating the well-known failure of black-box reward models that focus on surface aesthetics.

Parameter- and compute-efficient fine-tuning (using LoRA or layer-only adaptations on strong VLM backbones) is shown to suffice for alignment, demonstrating dataset efficiency and extensibility.

Empirical Findings: Metric Failures and MIE Gains

Rigorous cross-metric evaluation on MIB-Gold demonstrates unequivocally that all existing metrics (CLIP, DINOv2, PickScore, ImageReward, HPS v2.1, SCR, even low-level metrics) collapse to random or worse as subject count and scene complexity increase. Notably, reward models actively misalign with human preference in multi-entity scenarios (PickScore: 0.486, HPS: 0.520 agreement; chance = 0.5).

MIE, trained solely on MIB-Silver, substantially outperforms all baselines:

• Overall pairwise accuracy: 0.922 on Gold, with 0.982 for seen generators and 0.884 for unseen models.
• Diagnostic macro-F1: 0.818, indicating robust capture of structured failure signals, not just global aesthetic cues.

  Figure 3: MIE exhibits consistently higher pairwise accuracy against human judgment and diagnostic F1, including in challenging unseen-generator regimes; the largest LoRA-layer-tuned evaluator achieves the highest human alignment.

Breakdown analysis elucidates that LoRA-layer adaptation yields clear improvements over layer-only at all backbone scales, and that large model capacity must be coupled with the right adaptation method for generalization, especially out-of-distribution.

Figure 5: LoRA-layer tuning consistently improves diagnostic granularity and seen-to-unseen generalization, especially for Appearance and Interaction dimensions.

Existence is the most stable diagnostic category, while complex appearance and interaction failures remain discriminative—reinforcing the necessity of multi-dimensional evaluation rather than monolithic scoring.

Annotator Observations: Failure Modes in Modern Generators

Direct human inspection exposes key systematic failures in all evaluated SOTA personalized image generators:

• Failure rate increases with subject count: Omission and identity confusion dominate in 6–8 subject scenes, often compounding with appearance drift.

  Figure 2: Models uniformly exhibit Existence failures with increasing subject count, ranging from partial to severe omissions.

• Existence-Appearance co-occurrence: When a subject is omitted, its visual features are likely to appear redistributed (“hybridized”) across present entities, causing diagnostic signals to intertwine.

  Figure 4: Omitted subjects' attributes are merged into surviving figures, leading to non-independent Existence and Appearance failures.

• Multi-action overload: Assigning a single subject multiple concurrent actions leads to subject duplication (splitting) or silent action dropout, directly exposing current limitations in compositional semantic control.

  Figure 6: Multi-action prompts induce either subject duplication (“figure splitting”) or neglect of actions (“action dropout”), seen even in better-performing models.

Practical and Theoretical Implications

MIBE advances the state-of-evaluation by enabling:

• Rigorous, reproducible comparison of personalized image generators in realistic high-entity, interaction-rich scenarios.
• Fine-grained ablation and diagnosis to guide architectural or optimization-centered improvements targeted at specific binding failures.
• Scalable supervision for reward-model and evaluator alignment, facilitating future research on more compositional, structured multimodal discriminators.
• Establishment of more robust model improvement objectives, as standard aggregate metrics are shown insufficient.

The factorized diagnostic paradigm can be extended to more abstract compositional domains (e.g., multimodal reasoning, language-vision-action tasks), where interpretable, dimension-specific evaluation is required for trustable and actionable model advancement.

Limitations and Future Directions

Acknowledged limitations include:

• Finite reference diversity and coverage, primarily in controlled white-studio domains; long-tail and in-the-wild scenarios remain unaddressed.
• Temporal drift as new models and architectures emerge; ongoing reannotation and extension protocols are planned to sustain benchmark validity.
• Noisiness in VLM-labeled Silver Set; human-in-the-loop auditing is recommended when leveraging silver supervision for sensitive applications.

Future developments may focus on extending MIB to open, unconstrained settings, expanding reference and prompt coverage, improving cross-cultural and domain generality, and facilitating continual metric/model co-evolution.

Conclusion

MIBE is established as the first comprehensive, human-aligned framework for robust evaluation and diagnosis of multi-subject personalized image generation. Through rigorous hierarchical design, dual-strata annotation, and dual-head evaluator modeling, it exposes and quantifies the multi-faceted nature of binding failures, enables high-fidelity cross-model comparison, and provides direct interpretability for generation research.

MIBE (benchmark and models) is released under permissive terms to catalyze community adoption, rigorous benchmarking, and further rapid progress on the binding problem. Its impact is expected to extend not just to image generation but to the broader ecosystem of reference-grounded multimodal AI.