Hierarchical Concept-to-Appearance Guidance

• Hierarchical Concept-to-Appearance Guidance (CAG) is a two-stage method that separates concept-level and appearance-level guidance to ensure semantic and fine-grained alignment in multi-subject image generation.
• It leverages VAE dropout, visual language models, and correspondence-aware masked attention to address identity inconsistency and improve compositional control.
• Empirical evaluations demonstrate significant gains in prompt following, subject consistency, and overall realism, highlighting CAG’s effectiveness in complex scene synthesis.

Hierarchical Concept-to-Appearance Guidance (CAG) is a two-stage, structured conditioning paradigm designed to enhance multi-subject image generation by enforcing explicit, hierarchical supervision from high-level semantic concepts down to fine-grained appearance attributes. Unlike conventional approaches that implicitly associate text instructions with reference images, CAG operates by disentangling and hierarchically coordinating conceptual and appearance constraints, leveraging visual LLMs (VLMs), variational autoencoder (VAE) feature dropout, and a correspondence-aware attention mechanism within a diffusion transformer. This explicit structure effectively addresses identity inconsistency and limited compositional control in complex, multi-entity scenes (Xu et al., 3 Feb 2026).

1. Motivation and High-Level Structure

CAG is motivated by two primary limitations of existing diffusion-based multi-subject generation systems: (1) Identity inconsistency, where entangled or missing appearance cues in reference images result in generated subjects diverging from their intended identities, and (2) insufficient compositional control, manifesting as imprecise subject arrangement or attribute mixing in response to detailed user instructions.

The CAG pipeline addresses these with a hierarchical separation:

• Concept-Level Guidance: Imposes semantic alignment between generated content and user intent, robust to missing low-level appearance cues.
• Appearance-Level Guidance: Enforces fine-grained, tokenwise alignment, explicitly binding prompt words to corresponding spatial regions and reference attributes.

At training, a VAE dropout mechanism disrupts reliance on explicit appearance features, compelling the backbone (Diffusion Transformer, DiT) to respect VLM-derived, semantically grounded concepts. During appearance refinement, attention masking driven by VLM correspondence ensures accurate word-to-region mapping.

2. Concept-Level Guidance via VAE Dropout

At the concept stage, reference subjects are encoded via a VAE to obtain latent tokens $z_{\text{ref}} \in \mathbb{R}^{T \times D}$, where $T$ is the total number of tokens across all $N$ references. To obviate overfitting and encourage semantic reliance, a binary mask $m \in \{0,1\}^{T \times 1}$ with i.i.d. Bernoulli($1-p$) entries is sampled per iteration, yielding masked latents $\tilde z_{\text{ref}} = m \odot z_{\text{ref}}$.

The DiT is conditioned on $$\{ \tilde z_{\text{ref}}, h_{\text{ref}}, h_{\text{text}} \}$$, where $h_{\text{ref}}$ and $h_{\text{text}}$ are global VLM-encoded features. To ensure conceptual alignment even when $z_{\text{ref}}$ is partially or completely dropped, training includes both the diffusion denoising loss and a semantic consistency penalty:

$$L_\text{stage1} = L_\text{diff} + \lambda_c L_\text{concept}$$

where

$$L_\text{diff} = \mathbb{E}_{t,\epsilon} [\| \epsilon - \epsilon_\theta(x_t \mid \tilde z_{\text{ref}}, h_{\text{ref}}, h_{\text{text}}) \|^2]$$

$$L_\text{concept} = \mathbb{E} [ \| F_\text{VLM}(I_\text{gen}) - F_\text{VLM}(I_\text{ref}) \|_1 ]$$

$F_\text{VLM}$ computes pooled semantic embeddings, penalizing semantic drift during VAE dropout. This strategy compels the model to leverage VLM cues, enhancing semantic robustness and compositional fidelity (Xu et al., 3 Feb 2026).

3. Appearance-Level Guidance with Correspondence-Aware Masked Attention

To achieve precise attribute and region-wise binding between prompt tokens and reference images, CAG introduces a masked attention mechanism. Editing instructions are parsed by the VLM into referential words $W = \{w_i\}$. For each $w_i$, the VLM supplies a tuple $(\text{ref\_id}_i, \text{bbox}_i)$ determining which reference and spatial region correspond to that token.

An attention mask $M \in \{0,1\}^{|W| \times R}$ is constructed, where $R$ is the total number of VAE reference tokens. Each row in $M$ determines which reference tokens are eligible to be attended to by each prompt token. The corresponding modified transformer self-attention at each layer is:

$$\mathrm{Attn}(Q_i, K, V)_i = \sum_j \mathrm{softmax} \left( \frac{Q_i K_j^\top + \alpha M_{i,j}}{\sqrt{d}} \right) V_j$$

where $\alpha$ is a logit-boosting hyperparameter. When $M_{i,j}=1$, cross-modal attention is promoted; when $M_{i,j}=0$, attention from $w_i$ to reference region $j$ is suppressed.

This explicit mapping ensures that, for example, text tokens pertaining to “Alice” and “red dress” precisely access appearance tokens of the correct regions, preventing attribute leakage between subjects and supporting detailed scene composition (Xu et al., 3 Feb 2026).

4. Training and Inference Objectives

CAG combines concept and appearance objectives into a joint loss:

$$L = L_\text{diff} + \lambda_c L_\text{concept} + \lambda_a L_\text{appear}$$

where

• $L_\text{diff}$ is always active,
• $L_\text{concept}$ is applied during VAE dropout,
• $L_\text{appear}$ regularizes masked attention (e.g., encourages high probability for ground-truth token-to-region mappings).

Optimization details include AdamW (learning rate $10^{-5}$, batch size $8$), VAE dropout probability $p=0.5$, text dropout for classifier-free guidance $0.1$, full DiT fine-tuning, and frozen VLM. Guidance weight $\sigma$ is annealed from $1$ to $4$ over the first $2$k steps. At inference, correspondence masks are computed via VLM, and $25$ diffusion steps are performed with guidance scale $4.0$, with masked attention applied in every layer (Xu et al., 3 Feb 2026).

5. Empirical Evaluation and Comparisons

CAG was evaluated on curated datasets with $\sim24$k multi-character, multi-scene training samples and $300$ held-out prompts, each featuring $2$–$3$ subjects and diverse scene contexts.

Key metrics include:

Metric	Definition	Increment vs. OmniGen2/Best Baseline
PF	Prompt following (0–10)	+2.66 / +1.42
SC	Subject consistency (0–10)	+1.86 / +1.29
PF·SC (geom. mean)	Overall	+2.33 / +1.39
FID	Global realism (↓ better)	12% reduction

Qualitative results confirm accurate spatial arrangements, faithful reproduction of fine appearance details (hair color, clothing texture), and robust support for diverse instructional viewpoints (e.g., low/high angle). Ablations validate each module: optimal VAE dropout at $p\approx 0.4$–$0.5$; removing correspondence masking degrades PF by $0.24$ and SC by $0.18$; inference without VAE tokens collapses all metrics for the baseline, but CAG remains robust (Xu et al., 3 Feb 2026).

6. Position Within Hierarchical Concept-to-Appearance Research

Hierarchical concept-to-appearance approaches are increasingly central in explainable diagnosis (e.g., the CoPA framework), compositional text-to-image generation (e.g., HiCoGen), and multi-subject control. CoPA exemplifies hierarchical extraction and injection of concept-level cues at all depths of a vision transformer, enforcing alignment with textual concepts and preserving interpretability (Dong et al., 4 Oct 2025). HiCoGen, conversely, decomposes prompts sequentially, using reinforcement learning with hierarchical reward, but does not explicitly ground word-region correspondences inside attention layers (Yang et al., 25 Nov 2025). CAG is distinct in its two-stage guidance, concept dropout, and attention masking.

7. Context and Prospects

CAG sets a strong empirical benchmark for identity-consistent, prompt-controllable image synthesis in complex multi-subject scenarios. Variants of hierarchical guidance, such as CoPA’s multi-scale embedding routing or HiCoGen’s chain-of-synthesis, suggest that further unification or extension—such as integrating multi-level concept extraction with per-token attention masking—may drive the next advances in compositional visual generation and interpretable vision systems. A plausible implication is that cross-domain generalization for both medical and general visual reasoning tasks will increasingly rely on such hierarchical, disentangled, and explicitly grounded architectures (Xu et al., 3 Feb 2026, Yang et al., 25 Nov 2025, Dong et al., 4 Oct 2025).